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ABSTRACT 

The Extended School Services (ESS) program was established in 
1990 as part of the Kentucky Education Reform Act. Designed specifically to 
address the needs of Kentucky' s at-risk student population, ESS is an 
aggressive, proactive program for addressing academic problems before they 
become ingrained. This report summarizes findings from 24 site visits to 
selected schools across Kentucky that were implementing a variety of ESS 
programs; A total of 32 conclusions were generated from the evaluation, 
divided into nine topical areas: student demographics, adherence to intended 
goals, classroom instruction, student outcomes, program strengths, barriers 
to maximum success, program fidelity, patterns of implementation, and 
overall. Based on the findings and conclusions of this comprehensive 
evaluation of the statewide Kentucky Extended School Services Program, a 
number of specific recommendations are offered for staff's review and 
reflection. These 12 recommendations focus on transportation, professional 
development, communication and goal setting, and staffing. Nine appendixes 
contain evaluation checklists, surveys, questionnaires, and forms. (Contains 
27 references and 17 tables.) (GCP) 
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EXECUTIVE SUMMARY 



The Extended School Services (ESS) program was established in 1990 as part of the 
Kentucky Education Reform Act. Designed specifically to address the needs of Kentucky's at- 
risk student population, ESS is an aggressive, proactive program for addressing academic 
problems before they become ingrained. The ESS program extends the school day, week, or 
year for students at risk of academic failure, providing them with additional instructional time to 
help them meet academic goals. All Kentucky school districts receive funding earmarked for 
ESS implementation. 

In 1999, the Kentucky Commissioner of Education contracted with Pamela Nesselrodt 
and Eugene Schaffer for an external evaluation of the ESS program, which was to be piloted in 
the spring of 2000 and conducted during the 2000-01 academic year. Nesselrodt and Schaffer 
completed a pilot-test evaluation of the ESS program in the spring of 2000 that resulted in two 
reports — one on the design, testing, and refinement of instruments and another on the refinement 
and finalization of research questions and methodology. The pilot-test evaluation yielded a data 
collection design, data collection procedures and instruments, and analysis procedures. 

In the fall of 2001, the Kentucky Department of Education (KDE) contracted with a 
partnership of AEL and Western Kentucky University (WKU) for a comprehensive evaluation of 
the ESS program during the 2001-02 school year. All learnings from the pilot-test evaluation 
were incorporated into AEL's evaluation design. Fifteen evaluation questions were assembled 
into five major categories: identification, referral, and assignment of services; profiles of 

students receiving services; profiles of ESS programs and their implementation patterns; services 
to students placed at risk; and outcomes of the program. AEL's comprehensive evaluation of the 
ESS program utilized two major components — statewide surveys and site visits. 

The surveys were administered to the district and school ESS coordinators in the fall of 
2001 (full results presented in an earlier report). The major purposes for the site school visits in 
the fall/winter of 2001-02 and the summer of 2002 were to provide intensive, extensive ESS 
program data that would be generalizable, valid, and reliable to Kentucky programs statewide 
and to corroborate findings from the statewide surveys. A pair of trained data collectors made 
two to three day visits to a sample of 24 schools with ESS programs (18 during the fall/winter of 
2001-02 and 6 in the summer of 2002) to collect both qualitative and quantitative data from a 
variety of ESS stakeholder groups. This report summarizes findings from the 24 site visits to 
selected schools across Kentucky that were implementing a variety of ESS programs, i.e., before 
school, after school, and summer school. 

Data collection methods included six surveys (district coordinator, school coordinator, 
ESS teacher, non-ESS teacher, ESS student, and parent of ESS student); five interview protocols 
(district coordinator, school coordinator, ESS teacher, ESS student, and parent of ESS student); 
the Special Strategies Observation System (SSOS), which included three forms related to 
classroom observation, quality of instruction, and environmental resources of the classroom; and 
a school and program description form. AEL staff also selected and prepared in scannable 
format two other data collection instruments. First was the AEL Continuous School 
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Improvement Questionnaire (AEL CSIQ), a 60-item self-report that measures the extent to which 
a school faculty is committed to continuous improvement. Second was the Innovation 
Component Configuration Map for Extended School Services (ICCM), which was developed by 
the Kentucky Institute for Education Research (KIER) in the mid-1990s. 

In October 2001, the AEL CSIQ instrument was administered to faculty members of the 
48 schools in the sample pool provided by KDE. Analyses were generated for each individual 
school and for aggregated building-level profiles for elementary, middle, and high schools. In 
March 2002, AEL staff designed and mailed a one-page summary report for each school’s 
results, which included descriptions of the six scales and a chart displaying the school’s results as 
compared to the composite scores for all sample schools at the same building level (i.e., 
elementary, middle, high). Also in October, introductory letters from the Kentucky 
Commissioner of Education were sent to the 18 selected schools (6 elementary, 6 middle, and 6 
high schools). All 18 visits were conducted by the end of February 2002. In April 2002, a 
conference call involving AEL and KDE staff was held to select the sites for the summer visits. 
The decision was made to revisit 6 of the original 18 schools (2 elementary, 1 middle, and 3 
high schools); all six visits were conducted during June 2002. 

Evaluation findings are presented in two distinct sections: first by the individual data 
collection instruments so that all data are available for interpretation and use and second by 
comprehensive summaries of data directly aligned to the five broad evaluation topics and their 
related subquestions. The findings resulted in more than 60 pages of text, including 14 tables 
and 20 figures. Data included observations from 137 regular classrooms and 76 ESS sessions; 
respondent groups included 1,220 ESS student surveys and 109 interviews; 576 parent surveys 
and 49 interviews; 225 ESS teacher surveys and 98 interviews; 15 ESS district coordinator 
interviews; 23 ESS school coordinator interviews; and 297 non-ESS teacher surveys. 

A total of 32 conclusions were generated from this evaluation. These conclusions were 
divided into nine topical areas: student demographics, adherence to intended goals, classroom 
instruction, student outcomes, program strengths, barriers to maximum success, program fidelity, 
patterns of implementation, and overall. Representative conclusions from each topical area are 
included below. 

> In terms of the proportion of ESS enrollment to student enrollment, ESS participation is fairly equal 
across elementary, middle, and high school building levels; however, participation varies widely at 
the individual school level. 

> The participation of boys and girls in ESS is roughly equivalent, particularly at the elementary level. 
However, fewer females participate in the program at the middle and secondary levels. This warrants 
further investigation to determine whether middle and high school girls need fewer ESS services or if 
they are simply less interested than boys in ESS participation. 

> There is a great deal of consistency among the perceptions of coordinators, teachers, and parents as to 
how students are referred to ESS; the majority believe that students are referred most often by 
classroom teachers. However, students report that they most often self-select into the program. It 
may be that students are taking credit for self-selection by agreeing to participate in this voluntary 
program after a teacher or parent has made the suggestion. Either approach seems to allow enough 
flexibility for the intended population to become involved with the program. 
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> The students’ regular teachers, ESS school coordinators, and ESS teachers most often determine 
individual student goals, with parents and students themselves being involved to a lesser extent. Thus 
students’ goals appear to be heavily influenced by their teachers, yet the majority of students adopt 
these goals as their own and appear to understand why they are expected to benefit from participation 
in the ESS program. 

> ESS and regular classrooms differ on two major dimensions: quality of instruction and appropriate 
level of instruction. Quality of instruction is better in regular classrooms, but instructional level is 
more often appropriate in ESS classrooms. 

> ESS classrooms tend to engage in student-led activities, often involving independent seatwork and 
pair seatwork. Thus a “typical” ESS classroom appears to be one in which students work 
independently on homework and/or make-up tests, receiving individualized instruction as needed. 
One strength of the ESS classroom arrangement is that students are receiving the one-on-one tutoring 
they need and have the opportunity to have concepts not mastered retaught to them. 

> The ESS program appears to be having an impact on student performance. Nearly all teachers and 
coordinators indicate that participation in ESS has led to increased academic achievement. Further, 
parents report increased understanding of subject material by their children, that their children are 
passing a particular subject, or that their children are now doing better in school. 

> Parents and students also report improved study skills and increased motivation to learn as a result of 
participation in ESS. Students appreciate having opportunities to make up or retake tests. This 
flexibility for students who either missed a test or performed poorly on a test indicates that value is 
placed on allowing students the opportunity to show what they have learned. 

> The major strengths of the ESS program focus on processes for its implementation and outcomes 
resulting from that implementation. For instance, process-linked supports include targeting students 
as early as possible, dedicated staff, student transportation, collaboration between teachers and 
coordinators, flexible scheduling, low teacher/student ratio, and individualized instruction. 

> One unique strength of the ESS program is its fluidity and flexibility. Student mobility is high 
throughout the program. As a particular problem arises, ESS allows for an immediate intervention 
that focuses on a specific need that can be addressed before it becomes chronic and long term. The 
program does not rely solely on the results of annual standardized test scores, which would slow 
down the process of identification, referral, and enrollment. 

> Student transportation is a major problem for some schools. The decision to use ESS funds to 
provide public transportation for students is determined by individual school and/or district policies. 
Because the majority of the ESS services offered during the regular school year occur after normal 
school hours, if bus service is not provided then parents must make transportation arrangements for 
their children. With the combination of parental work schedules, a potential lack of transportation for 
lower-income families, and the distance involved for more rural communities, this factor could 
seriously deter participation of some students who might be most in need of such academic services. 

> Staff development related to ESS now seems to be nonexistent, inadequate, or distributed unevenly 
between teachers and coordinators. This may be more problematic for newer staff members who are 
initially becoming involved with ESS and who are not familiar with its related philosophies and 
guidelines, especially since the ESS summer conference was discontinued. Further, there is some 
lack of agreement among school coordinators, teachers, parents, and students as to the exact intent 
and nature of the ESS program. 

0 viii 
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> One discrepancy noted among respondent groups involves communication, especially with parents. 
While ESS teachers believe they meet with parents on an as-needed basis, parents note that 
communication with the teacher about their children’s progress is a major problem and that they often 
are not aware of ESS goals. 

> Most of the respondent groups believe that the current number of teachers involved in ESS is 
inadequate for the number of students. Related to this topic is the reported difficulty associated with 
recruiting, hiring, and retaining a sufficient number of interested teachers with appropriate content 
knowledge and relevant skills for working individually with students in the ESS environment. 

> The ESS programs are performing satisfactorily in terms of implementing the majority of the 15 

major components of the statewide program. The following four components seem to be 
implemented least satisfactorily: staff selection, instructional resources, collaborative planning 

processes, and program evaluation. 

> There are four types of implementation of ESS programs in terms of their fidelity in operating the 15 
major components of the program. That is, there are four levels of implementation of the ESS 
program, ranging from high-fidelity implementers to low-fidelity implementers. However, these 
patterns of implementation are very similar across the four groups; the main differences are in the 
levels of implementation of each component, as opposed to the differences across the components. 
Three of the four high implementation schools are middle schools with small ESS programs in terms 
of the number of involved students and teachers. In other words, the high-fidelity implementation is 
more an artifact of program scale and building level rather than discrete differences in 
implementation. 

> Although there seem to be no discemable operational differences in the four levels of implementation, 
there are some differences in associated measures when compared by implementation pattern. The 
high implementation group consistently spent less time on teacher-led activities and more time on 
student-led instructional activities than any of the remaining three groups. 

> When looking at implementation patterns with other data measures utilized in this comprehensive 

evaluation, one other conclusion can be drawn: All the ESS school coordinators in the high 

implementation group pinpointed inadequate financial support. 

> One of the most striking conclusions from this comprehensive evaluation of the statewide Kentucky 
Extended School Services program is the marked consistency and high degree of corroboration both 
within and among respondent perceptions and data collector observations. 

> Overall, it is concluded that the ESS program is positively perceived by involved stakeholders and 
has been proven to help address the needs of students who are at risk academically. However, several 
areas have been identified in which improvements could be made for a more successful 
implementation of the statewide program. 

Based on the findings and conclusions of this comprehensive evaluation of the statewide 
Kentucky Extended School Services program, a number of specific recommendations are offered 
for KDE staffs review and reflection. These 12 recommendations focus on transportation, 
professional development, communication and goal setting, and staffing. Representative 
recommendations are presented below. 
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y KDE staff and state board of education members should collaborate to identify possible 
solutions to transportation issues. Solutions might include working closely with 
transportation staff, investigating alternative funding formulas such as using non-ESS monies 
for transportation expenses and/or seeking additional funds specifically for transportation. 

y Professional development opportunities should be provided to ESS coordinators and teaching 
staff in the areas of staff selection, instructional resources, collaborative planning processes, 
individualized instruction, mentoring/tutoring, and program evaluation. The specific format 
for these professional development opportunities could vary from workshop sessions at a 
central site or decentralized sites to online, Internet-based courses. Whatever delivery 
method is selected, professional development in these four areas is needed by most ESS 
program staff in the state. 

y ESS staff should encourage/facilitate more involvement of parents and students in setting 
goals for individual students. This would help to improve communication between the home 
and school and to ensure that all involved parties share similar goals for individual students’ 
learning — further increasing the likelihood that these goals will be uniformly sought, 
supported, and achieved. In addition, continued communication with parents about their 
children’s progress should be a routine part of ESS program operation. 

y Some thought should be given to exploring ways to overcome the teacher staffing issue. For 
example, KDE staff could identify those districts experiencing ESS teacher recruitment 
problems and work with them to develop solutions. If the problem is teacher pay for ESS 

,• sessions and state or local regulations that prevent increasing teacher salaries, perhaps KDE 
staff could be instrumental in finding ways to overcome those barriers, such as seeking 
waivers for current rules or regulations. 

y The possibility of developing an incentive program for ESS teachers that would generate 
opportunities for recognition of their efforts should be investigated. For example, an ESS 
Teacher of the Year award program might be designed and implemented. The idea is to offer 
a significant award and possibly a financial reward based on state-established criteria. The 
award, which could be regional or statewide, may help draw teachers previously uninterested 
in participating in the ESS program. 

y KDE staff should formalize and fund the process for obtaining ESS “best practices” and 
develop a resource tool that would be available to all ESS staff. ESS staff in one or more 
districts could be financially compensated for spearheading the initiative and gathering 
submissions from all ESS programs. The final product could be in print or electronic format 
and would be a compendium of innovative and creative ESS programs. It could also include 
a segment on student motivation, as mentioned earlier. We understand that such an effort is 
currently under way, but statewide coordinators indicated limited awareness of this 
undertaking. Therefore, at the very least, KDE staff should increase the visibility and 
potential utility of such a tool for the ESS program statewide. One potential resource is the 
Promising Practices in Afterschool (PPAS) Web site, which provides detailed descriptions of 
promising practices nationwide (see www.afterschool.org). 
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INTRODUCTION 



Background 

The Extended School Services (ESS) program was established in 1990 as part of the 
Kentucky Education Reform Act (KERA). Designed specifically to address the needs of 
Kentucky's at-risk student population, ESS is an aggressive, proactive program for addressing 
academic problems before they become ingrained (Nesselrodt & Schaffer, 2000b). The ESS 
program extends the school day, week, or year for students at risk of academic failure, providing 
them with additional instructional time to help them meet academic goals. Rather than being an 
“add-on” or “stand-alone” program, ESS is designed to be an integral part of each school's 
regular academic program, thus ensuring that students receive instructional assistance in core 
content subjects in which they are performing poorly. 

The major emphases of the statewide ESS program are to (1) sustain students’ present 
level of performance to prevent them from falling behind; (2) provide extended programming for 
students who have been retained or are at risk of (a) being retained in a class or grade or 
(b) failing to graduate on time without assistance; and (3) close the achievement gap of low- 
performing students so they will perform successfully in the program appropriate to their age. 
All Kentucky school districts receive funding earmarked for ESS implementation. According to 
publications from the Division of Extended Learning of the Kentucky Department of Education 
(KDE), nearly every school provides such services; thus, nearly 1,450 schools have some type of 
ESS program (AEL, 2001; Quality Education Data, 1998). See the Kentucky ESS Web site for 
more detailed information (http://www.kde.state.ky.us/osle/extend/ESS/default.asp). 



Past Evaluations 

To date, three major within-state (internal) evaluations of the ESS program have been 
completed — one by the University of Kentucky in 1991, one by the KDE in 1993, and one by the 
Joint Center for the Study of Educational Policy at the University of Kentucky and the 
University of Louisville in 1998. In 1999, the Kentucky Commissioner of Education called for 
an external evaluation, i.e., by an agency(ies) outside of Kentucky. This evaluation was to be 
piloted in the spring of 2000 and conducted during the 2000-01 academic year (Nesselrodt & 
Schaffer, 2000b). 

The KDE approved a plan submitted by Drs. Pamela Nesselrodt and Eugene Schaffer (of 
Dickinson College, Pennsylvania, and the University of Maryland, respectively), which focused 
on four major categories related to the ESS program: (1) identification, referral, and assignment 
of services; (2) profiles of students receiving services; (3) profiles of ESS programs; and (4) out- 
comes of the programs (Nesselrodt & Schaffer, 2000a). The evaluators recommended using a 
variety of data collection procedures, including written surveys of multiple groups, interviews 
with samples from those groups, written program descriptions, classroom and ESS session 
observations, analysis of standardized achievement test scores, and statistical analysis of 
outcomes data. 
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Nesselrodt and Schaffer completed a pilot-test evaluation of the ESS program in the 
spring of 2000 that resulted in two reports — one on the design, testing, and refinement of 
instruments and another on the refinement and finalization of research questions and 
methodology. The pilot-test evaluation yielded a data collection design, data collection 
procedures and instruments, and analysis procedures. 



Current Evaluation 

In the fall of 2001, KDE contracted with a partnership of AEL and Western Kentucky 
University (WKU) for a comprehensive evaluation of the ESS program during the 2001-02 
school year. All learnings from the pilot-test evaluation were incorporated into AEL's evaluation 
design. Fifteen evaluation questions were assembled into five major categories: (1) identifica- 
tion, referral, and assignment of services; (2) profiles of students receiving services; (3) profiles 
of ESS programs and their implementation patterns; (4) services to students placed at risk; and 
(5) outcomes of the program. 

AEL's comprehensive evaluation of the ESS program utilized two major components- 
statewide surveys and site visits. The surveys were administered to the district and school ESS 
coordinators in the fall of 2001. For detailed results of the statewide survey administration, see 
Perceptions of Kentucky’s Extended School Services Program by District and School 
Coordinators (Cowley & Meehan, 2001). These components were broken down into five main 
phases: statewide surveys, training session for site visits, fall/winter site visits, summer visits, 
.and data analyses. See Figure 1 for a graphic portrayal of AEL’s evaluation of the ESS program 
and Figure 2 for a time line of major events and activities. 

The major purposes for the site school visits in the fall/winter of 2001-02 and the summer 
of 2002 were to provide intensive, extensive ESS program data that would be generalizable, 
valid, and reliable to Kentucky programs statewide and to corroborate findings from the 
statewide surveys previously administered to district and school ESS coordinators. The site 
visits replicated most of the procedures and data collection instruments utilized in the pilot-test 
evaluation, with modifications as described later in this report. A pair of trained data collectors 
made two to three day visits to a sample of 24 schools with ESS programs (18 during the 
fall/winter of 2001-02 and 6 in the summer of 2002) to collect both qualitative and quantitative 
data from a variety of ESS stakeholder groups.* This data collection involved classroom and 



*A two-stage sampling process was implemented to identify the 24 schools. In the first stage, KDE staff established a 
pool of 48 schools through a six-step process that reviewed student achievement data, percentage of students eligible 
for free or reduced-price meals, overall academic student index, ethnicity, school-level performance indicators such as 
novice-level readers and dropout rates, comparisons of subsets of student scores within schools, and geographic and 
demographic representations (see Appendix A for the KDE school selection process). AEL completed the second stage 
by securing Johnson locale codes (National Center for Education Statistics, 2001) and published enrollment figures 
(Quality Education Data, 1998) for each of the 48 schools. Using a combination of building level, geography, 
urbanicity, and enrollment, AEL staff selected the 18 schools for the fall/winter 2001-02 visits. AEL staff collaborated 
by telephone with KDE staff to identify which of the 1 8 schools would be revisited during the summer of 2002, based 
on geography, building level, and general representativeness of Kentucky ESS programs. 
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Fall/Winter Site Visits 

• 18 School Sites with ESS 

• Surveys: 

• ESS Teacher 

• Non-ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Interviews: 

• District ESS Coordinator 

• School ESS Coordinator 

• ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Classroom Observations: 

• SSOS Observation Form 

• QAIT Classroom Rating 

• CER Resources Checklist 

• School and Program Description Form 

• ICCM Form 



Final 
Report of 
the ESS 
Evaluation 



Summer Site Visits 

• 6 of 18 Sites Above 

• Same Surveys, Interviews, 

Observations, and Forms less the 
District ESS Coordinator Interview 
and the Non-ESS Teacher Survey 



Figure 1: Graphic Portrayal of the Evaluation 
of the Kentucky Extended School Services Program 
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Figure 2: Time Line and Flow of Major Events in the 
Evaluation of the Kentucky Extended School Services Program 
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ESS session observations; interviews with ESS teachers, ESS students, ESS parents, the school 
ESS coordinator, and the ESS district coordinator; surveys of non-ESS teachers, ESS teachers, 
ESS students, and ESS parents; a school and program description form; and written 
documentation such as the school's consolidated plan and needs assessment, as well as 
descriptions/policies of the ESS program. In addition, AEL added two new instruments — the 
Innovation Component Configuration Map, to generate patterns of implementation across ESS 
programs, and the AEL Continuous School Improvement Questionnaire, to measure the extent to 
which a school faculty is committed to continuous improvement. See Appendix B for a copy of 
the completed Evaluation Standards Checklist, which reflects the extent to which this evaluation 
met the Program Evaluation Standards (Joint Committee on Standards for Educational 
Evaluation, 1994). 



Purpose and Audience 

This report summarizes findings from the 24 site visits to selected schools across 
Kentucky that were implementing a variety of ESS programs, i.e., before school, after school, 
and summer school. Data from this comprehensive evaluation are intended to inform KDE staff 
about how students are identified and referred for ESS services, what types of students receive 
services, particular ESS implementation patterns, services provided to at-risk students, and 
outcomes of ESS programs. The main audiences for this report are KDE staff, Kentucky state 
board of education members, and Kentucky district and school ESS coordinators. Secondary 
audiences include other individuals or agencies interested in extended school services and/or 
helping students achieve academic success through nonregular instructional services. 



Review of Literature 

The use of extended school services has dramatically increased over the past decade. The 
focus of after-school programs has moved from simple child care to the provision of a variety of 
services: improving academic achievement, securing a safe and drug-free environment, 

extending learning time, and supporting social development and school reform. According to the 
National Governors Association (1999), at least 26 states are increasing funding for after-school 
programs, and at least 30 states are seeing greater school involvement in extended learning 
during the after-school hours. In a 2001 National Association of Elementary School Principals 
survey, more than two thirds of the principals reported they currently offer optional after-school 
programs (Noam, 2002). 

Several after-school programs began recently, such as the national 21 st Century 
Community Learning Centers (CCLC), California’s After School Learning and Safe 
Neighborhoods Partnership Project, and Maryland’s After School Opportunity Fund (Miller, 
2001). These multipurpose programs are intended to increase academic achievement but also 
focus on providing an environment that meets multiple needs of students. 
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After-school programs are quickly gaining public support: “Polls conducted for Fight 
Crime: Invest in Kids show that 68% of Americans say expanding child care and after-school 
programs is a higher priority than a tax cut” (Newman, Fox, Flynn, & Christeson, 2001, p. 25). 
Results of a 2001 survey indicate that more than 90% of American voters believe youth “should 
have organized activities or places to go after school every day that provide opportunities to 
learn” (Noam, 2002, p. 1). The funding for U.S. Department of Education’s 21 st Century 
Community Learning Centers grants has increased from $1 million in 1997 to $1 billion in 2002 
and will provide funding for approximately 6,800 schools across the nation (U.S. Department of 
Education, 2002). 

According to Trousdale (2000), “these programs have the potential not only to keep 
children safe and out of trouble, but also to provide engaging environments that motivate and 
inspire learning outside the regular school day” (p. 2). She states that research shows that 
student participation in after-school activities helps them spend less time watching television, 
and more time on new skills and interests. It also improves school attendance and homework 
completion and leads to higher aspirations for the future. This research suggests that investing in 
successful after-school programs will yield extremely high returns on several levels. 

Developing programs that offer high returns requires planning, commitment, and strong 
collaboration among all involved. While most researchers believe that promising models for 
after-school programs exist, it is still unclear which are the most effective (Fashola, 1998). Witt 
(2001) mentions several reports that identify elements essential for successful programs: “age- 
appropriate learning activities, a low student-staff ratio, qualified staff, linkages with the regular 
school program and with community, organizations, safety, a wide range of both structured and 
unstructured activities, program evaluation, and parent involvement” (p. 43). 

While there is a need for these multipurpose after-school programs, the Kentucky 
Extended School Services program - focuses solely on helping struggling learners. The ESS 
program is intended to intervene as soon as problems arise, rather than waiting until students are 
at risk of failing. The sessions are purposeful and guided by structured curricula. ESS is not 
viewed as a stand-alone program but as an extension of the regular classroom program (Council 
of Chief State School Officers, 2000). 

Two major challenges faced by both types of after-school programs are funding and 
qualified staff. To sustain funding, Lamer, Zippiroli, and Behrman (1999) recommend that 
schools develop a model that incorporates affordable parent fees, private support and donations, 
and increased government funding. Balancing these sources is required to ensure that programs 
are accessible to all. 

To maintain qualified staff, schools must invest in efforts that concentrate on the skills 
required for professional work with school-age children, such as training courses, degree 
programs, and an increase in compensation linked to improved qualifications (Lamer, Zippiroli, 
& Behrman, 1999). Staff must commit to supporting both the goals of the program and the goals 
of the participating students. 
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Only within the past few years has research documenting the impact of after-school 
programs begun to accumulate. Trousdale (2000) reports that findings show slight improvement 
in students’ GPA after one year of program implementation. Grossman et al. (2002) found that 
students who participate in after-school programs are more likely to stay of out trouble during 
out-of-school time. Although they did not find significant improvements in grades or test scores, 
they did find indicators of academic improvement such as student’s effort, competency, pride 
in belonging to their school, and attentiveness. Parents also reported that their children who 
attend the programs try harder in school. Other child development and education studies suggest 
after-school participation is associated with better grades, peer relations, emotional adjustment, 
and peer resolution skills. Participating youth are involved in more learning opportunities, 
academic activities, and enrichment activities than those who do not participate (Noam, 2002). 

With the No Child Left Behind Act we see a surge in funding for after-school programs. 
Miller (2001) states, “after school programs seem to be the latest silver bullet solution to social 
and educational challenges, but support will be short-lived unless programs meet expectations” 
(p. 5). Two streams of thinking are the multipurpose programs, such as California’s After 
School Learning and Safe Neighborhoods, and more narrowly focused programs, such as 
Kentucky’s Extended School Services. If after-school programs are to succeed, we must provide 
them with sustainable funding, qualified staff, and the models to become highly effective 
programs that create meaningful and rich environments to engage and teach children. 
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METHODS 



The first task for this comprehensive evaluation was to convert all of the instruments used 
in the Nesselrodt and Schaffer pilot test (2000a, 2000b) to a machine-scannable format 
(excluding the interview protocols) and to make improvements based on lessons learned from the 
pilot test and suggestions from KDE staff. These instruments included six surveys (district 
coordinator, school coordinator, ESS teacher, non-ESS teacher, ESS student, and parent of ESS 
student); five interview protocols (district coordinator, school coordinator, ESS teacher, ESS 
student, and parent of ESS student); the Special Strategies Observation System (SSOS), which 
included three forms related to classroom observation, quality of instruction, and environmental 
resources of the classroom; and a school and program description form. AEL staff also selected 
and prepared in scannable format two other data collection instruments. First was the AEL 
Continuous School Improvement Questionnaire (AEL CSIQ), a 60-item self-report that measures 
the extent to which a school faculty is committed to continuous improvement. Second was the 
Innovation Component Configuration Map for Extended School Services (ICCM). This 
instrument was developed by the Kentucky Institute for Education Research (KIER) in the mid- 
1990s; AEL obtained permission from KIER to use the instrument in this evaluation. This first 
task was completed by August 2001 and all instruments were submitted to the AEL Institutional 
Review Board (IRB) for approval. A few minor changes were made in response to IRB 
suggestions and all instruments were approved for use in the evaluation. 



Statewide Surveys 



For a complete report on the instrumentation, data collection and analysis, and findings 
for the statewide coordinator surveys, see Perceptions of Kentucky ’s Extended School Services 
Program by District and School Coordinators (Cowley & Meehan, 2001). See Appendix C for a 
copy of each of the two statewide surveys. 



Training Session for Site Visits 



Planning for the training session began shortly after the contract was signed in August 
2001 and continued until the session took place. An experienced data collector was identified to 
conduct the actual training. This consultant was hired in September and was especially 
instrumental in helping revise the SSOS instrument. 

With the data collection instruments approved for use and a trainer on board, the next 
steps turned to designing the training session, developing the training manual, and hiring data 
collectors. The training session was designed as a three-day event to be held at AEL’s 
headquarters in Charleston, West Virginia. Data collectors included six AEL staff, four 
experienced Kentucky educators/consultants, and two West Virginia consultants. 
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A training manual was developed for data collectors to use during and after the training 
session. This manual contained copies of each instrument to be used in the evaluation along with 
instructions for administration. It also included an agenda, a participant list, an overview of the 
ESS project and evaluation, sections for each of the major types of data collection, procedural 
information (random selection information, student consent forms, site visit procedures, 
materials checklist), a calendar for scheduling site visits, and an evaluation form for the training 
session. In addition to using the training notebook for reference while actually conducting site 
visits, an e-mail listserv was developed so that data collectors could interact quickly and easily 
with one another and with AEL evaluation staff during the evaluation project. 

The training session took place on October 3-5, 2001. Twelve individuals completed the 
three-day training. On hand to welcome and greet the participants were the director of AEL’s 
Regional Educational Laboratory and the director of Kentucky’s Extended Learning Division. 
The bulk of the training time was devoted to the three instruments in the SSOS. Discussion of 
codes, practice with classroom videotapes, and instructions proceeded on schedule. One unique 
aspect of the training was the inclusion of actual paired practice observations conducted in 
classrooms at a nearby high school. After each live classroom observation during the second and 
third days of training, a debriefing session was conducted to discuss questions and concerns. 
The training also dealt with other instruments such as interviews and surveys. The AEL director 
of evaluation conducted the sessions on the ICCM and AEL CSIQ. 

Several wrap-up activities were completed at the conclusion of the training session. One 
was an evaluative activity in which participants coded a criterion tape of classroom behavior 
previously selected by the trainer, who reviewed each completed SSOS and verified that all 
participants met or exceeded the established level of competency in terms of coding 
specifications. A second wrap-up activity was making tentative pairings of data collectors and 
schools, based on interests, schedules, and geographical proximity. The third and final wrap-up 
activity was the completion of an evaluation form by all participants. 

When participants were asked to complete the evaluation forms, 1 1 individuals were on 
hand (one person had to leave early). The 14 items included a 1 to 5 response scale (very 
dissatisfied to very satisfied). As shown in Table 1, all 14 items had mean scores above 4.0, 
indicating a high level of satisfaction with the training content, process, facilities, and knowledge 
gained. The three items with the highest means were tied at 4.91 and included the 
resources/information gained for future use, the availability of needed equipment, and the 
training room accommodations; these three items also had the lowest standard deviation of 0.30. 
The lowest-rated item, with a mean of 4.22, was that the training session had stimulated the 
participants to change their current work behaviors. However, that particular item also had the 
largest standard deviation of 1.30, indicating a wide range of participant responses. Participants 
responded to four open-ended items. In the first, they indicated they liked the wonderful “risk- 
free” environment, the classroom observation practice, and the trainer’s knowledge and style. In 
the second, they indicated they learned how to use the forms, especially the SSOS. One 
respondent noted, “A completely new way of looking at classrooms as well as an important 
‘piece’ of the reform movement with which I have not been involved”; another said, “How to 
incorporate all observation instruments into a cohesive whole.” In the third item, participants 
suggested receiving the manual prior to the training and more practice with classroom 
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observations. In the fourth item, participants noted they needed time to think about and absorb 
the manual contents before asking for clarification on specific items. 

Table 1: Descriptive Statistics from the 
Data Collectors Training Session Evaluation Form 



Items 


N 


Mean 


Std. Dev. 


la: Amount of information 


11 


4.82 


0.60 


lb: Comprehensiveness of information 


11 


4.73 


0.90 


lc: Usefulness of information 


11 


4.82 


0.60 


Id: Technical quality of information 


11 


4.55 


0.69 


le: Potential to improve my work practice 


10 


4.70 


0.48 


2a: Presentation style 


11 


4.82 


0.60 


2b: Presentation efficiency 


11 


4.45 


1.21 


3a: Equipment availability 


11 


4.91 


0.30 


3b: Room accommodations 


11 


4.91 


0.30 


4: Provided me with information that I 


11 


4.91 


0.30 


can access for future use 








5: Increased my knowledge relative to the 


11 


4.82 


0.40 


topic presented 








6: Increased my skills relative to the topic 


11 


4.73 


0.47 


presented 








7: Provided me with knowledge and/or 


11 


4.82 


0.40 


skills to incorporate into my work 








8: Stimulated me to change my work 


9 


4.22 


1.30 


behavior to include new knowl./skills 








1: Hiked: 


The risk-free environment; t 
observations; and the trainer 
teaching style, experience, ai 


le classroom 
’s knowledge, 
id enthusiasm. 


2: I learned: 


New way of coding classrooms, how to use 
the various forms, especially the SSOS. 


3: Suggested improvements: 


Receive material before workshop, more 
time to practice, more classroom 
observations, more explanation of 
instruments. 


4: Still need clarification on: 


Need to think about all the information and 
practice more. 
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AEL CSIQ Administration 



Instrumentation. The AEL Continuous School Improvement Questionnaire (AEL 
CSIQ) is a 60-item instrument that measures the extent to which a school’s faculty members are 
committed to continuous improvement. The 60 items are each rated on a Likert-type scale of 
1 to 6 ( not present to present to a high degree ). These items comprise six scales: (1) learning 
culture, (2) school/family/community connections, (3) shared leadership, (4) shared goals for 
learning, (5) purposeful student assessment, and (6) effective teaching. This instrument was used 
to discern the connection between faculty members’ commitment to continuous improvement and 
implementation of their ESS programs. See Appendix D for a copy of this form. 

Data collection. After the training session was conducted, attention was given to 
administering the AEL CSIQ to faculty members of the 48 schools in the sample pool. During 
the second week of October, AEL staff drafted a letter for the Kentucky Commissioner of 
Education’s signature, soliciting cooperation from the 48 schools in this step of the evaluation. 
KDE staff returned the signed letter, printed 50 copies on letterhead, and mailed them to AEL. 
AEL staff then assembled kits containing the cover letter, the instruments, and a postage-paid 
return envelope; these kits were mailed on October 15. Schools continued to submit completed 
forms through the end of December. AEL staff called all non-responding schools and spoke with 
the principal or school coordinator to solicit information regarding the status of the instrument 
completion. Several schools requested replacement kits, which were prepared and mailed. In 
late January, KDE staff contacted the few remaining schools. By the end of February, 47 of the 
48 schools had returned completed forms, for a return rate of 98%. 

Data analysis. AEL staff scanned the surveys using Remark optical scanning software, 
cleaned the data files, and exported them to a standard software program (Statistical Package for 
the Social Sciences, now known as SPSS) for statistical analyses. Analysis was conducted at the 
scale level, and appropriate descriptive statistics were generated, i.e., means and standard 
deviations. Analyses were generated for each individual school and for aggregated building- 
level profiles for elementary, middle, and high schools. 

AEL staff designed a one-page summary report for each school’s results, which included 
descriptions of the six scales and a chart displaying the school’s results as compared to the 
composite scores for all sample schools at the same building level (i.e., elementary, middle, 
high). On March 22, 2002, AEL staff mailed these summaries to the schools, along with a cover 
letter thanking staff for their cooperation. 

AEL CSIQ data were also analyzed to determine whether significant differences existed 
by ESS implementation patterns. Descriptive statistics and one-way Analysis of Variance 
(ANOVAs) were utilized at the scale level for this analysis.* 



*Random sampling was not utilized in this evaluation, given the original sample pool of 48 schools provided by 
KDE. Therefore the assumption for the use of ANOVAs and t tests does not apply in a classical sense. However, 
they are informative for comparing within-group variance to between-group variance for some of the measures used 
in this evaluation. 
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Validity and reliability. The validity of this instrument was established in prior research 
conducted by AEL (Meehan, Cowley, Craig, Balow, & Childers, 2002). As a measure of the 
internal consistency reliability of the instrument for this administration, Cronbach Alpha values 
were computed for the 10 items within each of the six scales, as well as for all of the items in the 
instrument. For this set of scores, values ranged from .89 for the learning culture scale to .94 for 
effective teaching; the overall value for all items was .98. 



Site Visits 



Following the October training, data collectors were paired and teams were assigned to 
specific schools to conduct the site visits, which were usually two to three days in length. To 
begin the process, AEL staff drafted another letter for the Commissioner’s signature, announcing 
to the 18 schools (6 elementary, 6 middle, and 6 high schools) that they had been selected. KDE 
staff mailed the letters on October 26, 2001, and data collectors began contacting schools to 
schedule their site visits. As expected, some juggling occurred before teams had final schedules; 
the e-mail listserv was extremely useful in finalizing plans. By the second week of November, 
all of the visits were scheduled for completion by the end of February 2002; most data collection 
teams were scheduled for three site visits. Of the original 18 schools selected for the fall/winter 
site visits, one was eventually replaced with a preselected alternate due to a scheduling conflict. 

In March 2002, AEL staff sent KDE staff an e-mail message to start the process of 
selecting six schools for the summer visits. On April 25, a conference call involving AEL and 
KDE staff was held to select the schools and two alternates. The decision was made to revisit 6 
of the original 1 8 schools (2 elementary, 1 middle, and 3 high schools) for the summer 2002 site 
visits. Such a strategy would provide data from the schools’ regular after-school ESS program 
and their summer programs. KDE staff contacted an administrator at each school and secured 
their participation. 

Data collectors were contacted and began scheduling summer site visits. All six visits 
were conducted during June 2002. In terms of methodology, the only differences in the summer 
site visits were the exclusion of the non-ESS teacher survey, the decision not to re-interview the 
district administrator, an increase in the targeted number of parent and student interviews, and an 
increase in the number of ESS students observed. As well, a few of the interview questions for 
parents, students, ESS teachers, and school coordinators were eliminated because they were not 
relevant to the summer programs. 

For all site visits, teams worked with the ESS school coordinators to finalize plans for the 
site visits and to implement all data collection activities. Given their proximity to their schools, 
the four Kentucky data collectors made half-day visits prior to the site visits to meet the 
coordinators, review the site visit processes, and request school materials. West Virginia teams 
relied on telephone and e-mail to schedule and make final arrangements. Teams gathered 
pertinent information from each school regarding the number of surveys and student consent 
forms needed, and AEL staff prepared and mailed packets. 
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At the conclusion of each site visit, data collectors returned materials. Each school was 
logged in and checked for completeness. Data collectors were notified of any missing materials; 
completed materials were filed and stored by school. All signed consent forms were sent to 
AEL’s Administrative Services Office for archival purposes. Interview notes were submitted to 
AEL support staff for typing before qualitative analysis began. 

The following sections describe the instrumentation, data collection, data analysis, and 
reliability information for each of the four major techniques utilized during the site visits: 
surveys, interview protocols, classroom observations, and other instruments. 



Surveys 

Instrumentation. Four survey instruments were developed, tested, and utilized in the 
ESS pilot test and employed during the site visits of this evaluation. The four target groups 
included ESS teachers, non-ESS teachers, ESS students, and parents of ESS students. Each 
survey is briefly described below. See Appendix E for a copy of each survey. 

• ESS teacher: This survey contained 24 questions on a folded 11" x 17" sheet. The initial 
18 items were selected-response and focused on demographics, attributes of the ESS 
program, recruitment procedures, teacher hiring and staff development, communication 
strategies, major outcomes, forces that help or hinder implementation, and overall 
effectiveness of the program. Six open-ended items asked for information related to 
strengths, weaknesses, and recommendations for improvements. This survey was 
administered at school to the full population of ESS teachers. 

• Non-ESS teacher: This survey contained 19 questions on a folded 11" x 17" sheet: 13 
selected-response and 6 open-ended items, all similar to those on the ESS teacher survey. 
This survey was administered at school to the full population of teachers not engaged in 
the ESS program. 

• Parent of ESS student: This survey contained 1 1 questions on a 8 14" x 11 " sheet. The 
initial 7 items were selected-response and focused on their children’s performance in 
ESS. Four open-ended items focused on best features or problems of the program and an 
explanation of why their children would or would not participate the following year. 
This survey was administered to parents of all students participating in the ESS program 
who had returned a signed parental consent form. One copy of the survey was sent home 
with the student and was completed by one parent individually or by both collaboratively. 

• ESS student: This survey contained 16 questions on one 8 14 " x 11" sheet. Thirteen of 
the items were selected-response and focused on demographics, subjects studied in ESS, 
and a series of questions about academic behaviors and attitudes with which students 
agreed or disagreed. Three open-ended items focused on what students liked best about 
the ESS program and what changes should be made to the program. This survey was 
administered at school to all students participating in the ESS program who had returned 
a signed parental consent form authorizing their participation in evaluation activities. 
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Data collection. During each site visit, the data collectors worked with the school 
coordinator to administer the four surveys. Surveys were generally distributed during the first 
day of the visit, and returned by the last day. In some cases, school coordinators offered to 
forward to AEL any surveys returned after the site visit; however, no other surveys were ever 
received by AEL staff. As noted earlier, there was wide variance among the participation rate of 
students, due to the fluctuating percentages of signed consent forms. Further, there was a 
marked lack of compliance among the non-ESS teachers in completing and returning their 
surveys. Given the above situations, along with the fluidity inherent in the ESS program, return 
rate percentages were not calculated for survey respondents. However, returns were deemed 
satisfactory for ESS teachers and ESS students; less satisfactory were responses from non-ESS 
teachers and parents of ESS students. 

Data analyses. After all of the fall/winter site visits were completed, AEL staff designed 
data entry templates using Remark optical scanning software. Surveys were scanned and then 
exported to SPSS for statistical analysis, including descriptive statistics. Although open-ended 
comments were originally scanned into the data files, this produced fairly unreadable pictures of 
the comments. Therefore, these comments were typed into the SPSS data files. Summer site 
visit data were scanned as they became available. Individual school files were merged into one 
master file by type of survey before analyses began. A breakdown of the number of surveys 
completed during the fall/winter and summer site visits is provided in Table 2. 

Table 2: Number of Site Visit Evaluation Survey Respondents by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


ESS teachers 


175 


50 


225 


Non-ESS teachers 


297 


NA 


297 


ESS students 


775 


445 


1,220 


Parents of ESS students 


360 


216 


576 


Totals 


1,607 


711 


2,318 



Validity and reliability. These surveys were developed, tested, and utilized in the pilot 
test by Nesselrodt and Schaffer (2000a, 2000b). Prior to their use in this evaluation, AEL and 
KDE staff made final revisions to all four surveys and converted them to a scannable format. 
Thus, these instruments possess face and content validity and have proven their utility in a prior 
administration. - To assess the degree of internal consistency reliability, Cronbach alpha 
coefficients were computed for this administration of the selected-response items on the four 
surveys, excluding demographic-type items and, for the parent survey, one particular multiple- 
response item. This administration of the ESS teacher survey resulted in a coefficient of .60; the 
non-ESS teacher survey, a .53; the parent survey, a .53; and the student survey, a .70. While 
some of these coefficients are lower than desired, when balanced with the need for face validity 
they do verify that the survey items, in general, relate to and contribute to the same construct. 
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Interview Protocols 

Instrumentation. Five interview protocols were developed, tested, and utilized in the 
pilot test and employed in this evaluation. These protocols provided the means to secure 
detailed, in-depth information about the topics covered in the surveys and yielded rich data 
regarding topics of interest. The five target groups included district administrators 
(coordinators), school administrators (coordinators), ESS teachers, ESS students, and parents of 
ESS students. Each protocol is briefly described below. See Appendix F for a copy of each 
interview protocol. 

• District and school coordinators: These protocols each contained 20 questions and 
focused on aspects such as the history of ESS programs in the district/school; a 
description of the services provided under the current program; how students are 
identified and referred to the program; recruitment and staff development for 
administrators and teachers; communication among teaching staff, students, and parents; 
processes for setting goals and monitoring student progress; and assessment and exit 
procedures. Both coordinators were interviewed during the fall/winter site visits; only 
the school coordinator was interviewed during the summer site visits. 

• ESS teacher: This protocol contained 16 questions and focused on aspects such as a 
description of the current ESS program, recruitment and staff development practices, 
communications with students and parents, curriculum and methodologies used in regular 
and ESS classrooms, and major strengths and weaknesses of the current ESS program. 
For the fall/winter visits, four ESS teacher interviews were planned; for the summer 
visits, five interviews were to be completed. 

• Parent of ESS student: This protocol contained 1 1 questions and focused on perceived 
effectiveness of the program, areas of potential change, and perceived growth or success 
of their children. Two parent interviews were planned for the fall/winter visits and three 
for the summer visits. 

• ESS student: This protocol contained 6 main questions, with multiple sub-items within 4 
of the main questions and focused on perceived effectiveness of the program, areas of 
potential change, and perceived growth or success.: Four student interviews were planned 
for the fall/winter visits and six for the summer visits. 

Data collection. During each site visit, arrangements were made to interview the district 
coordinator at either the district office or the school. With the exception of parents, all 
interviews were conducted during the school day. With the exception of students, all interviews 
were conducted on an individual basis. For the student interviews, it was decided that younger 
students might feel more comfortable and willing to express their views in the company of a 
peer; therefore, elementary and middle-level students were interviewed in pairs. Extensive 
interview notes were taken, which were later transcribed for analysis purposes. Students, 
parents, and teachers were randomly selected for interviewing. For students, a random selection 
was made using the signed parental consent forms received by the school coordinator. In 
practice, this return step worked less than perfectly, with varying return rates across sites of the 
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signed consent forms, which led to limited student names for random sampling. Parents were 
randomly selected from the pool of students to be interviewed. For teachers (where there were 
more than the desired number), a random selection was made from a listing of names. 

Data analyses. By the end of the summer site visits, all interviews had been completed. 
A breakdown of the number of interviews completed during the fall/winter and summer site 
visits is provided in Table 3. 



Table 3: Number of Site Visit Evaluation Interview Participants by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


District coordinators 


15* 1 


0 


15 


School coordinators 


17 ** 


6 


23 


ESS teachers 


72 


26 


98 


ESS students 


74 


35 


109 


Parents of ESS students 


36 n 


13 


49 


Totals 


214 


80 


294 



*Three of the district coordinators had two schools within the 18 schools selected for site visits. 
**One of the school coordinators was unavailable due to an injury suffered the day before the 
site visit. 



An AEL staff member analyzed all the fall/winter interview responses that directly 
related to the five main evaluation questions for each of the five groups of interviewees. A first 
reading of the responses led to a rough cut of similar answers among respondents. A second 
reading discovered emerging themes. Tables were drafted for each interview question to display 
the response percentages in each category, the category names, and a sample of representative 
responses. These tables were later updated to include responses generated during the summer 
interviews. 

Validity and reliability. These protocols were developed, tested, and utilized in the pilot 
test by Nesselrodt and Schaffer (2000a, 2000b). Prior to their use in this evaluation, AEL and 
KDE staff made final revisions to all five protocols. Thus, these protocols possess face and 
content validity and have proven their utility in a prior administration. For reliability, 
interviewers’ use of protocols at the training session and during the site visits established a 
satisfactory level of agreement. 



Classroom Observation Forms 

Instrumentation. For the collection of regular classroom and ESS session data, three 
collection instruments were selected and employed in the pilot test and used in this evaluation. 
All three were developed and employed in prior research and evaluation studies, refined by AEL 
and KDE staff, and converted to a scannable format. The three instruments comprise the Special 
Strategies Observation System (SSOS), which is designed for use in a variety of settings to 
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systematically collect data on essential elements of classroom behavior related to instruction, 
management, and context. The SSOS is a viable instrument for school effectiveness research 
due to its strong grounding in the current literature on effective teaching and its utilization of a 
variety of methodologies. This combination of instruments generates low-, moderate-, and high- 
inference data; this triangulation of data sources further documents the veracity of the data 
collected. Each instrument that makes up the SSOS is described below. See Appendix G for a 
copy of the SSOS form. 

• Classroom observation (also called SSOS): The SSOS is a combination observation 
system that is best described as a category system, with both low and high inference 
items, and includes multiple coding procedures (Nesselrodt & Schaffer, 1993; Sullivan & 
Meehan, 1983). It is based on the Classroom Activity Record designed by Everston and 
Burry (1989) and the Stallings Observation System (Stallings, 1980). The top page of the 
SSOS collects typical demographic information, such as the school, observer, date, 
number of adults and students in class, subject being observed, and type of class (ESS or 
regular). The observations occur over 56 minutes, during which the observer switches 
between coding the entire classroom and focusing on a single student previously selected. 
Each of seven pages corresponds to eight minutes of class time. The first minute per 
page looks at student engagement (i.e., the number of students on task, off task, out of the 
room, or waiting) and grouping strategies (i.e., whether clustered in teacher, aide, or 
student groups and type of involvement, such as working alone, management, interaction, 
or socialization). The remaining seven minutes per page focus specifically on the target 
student, and include coding one of 27 discrete activities for each minute. 

Analyses of the completed SSOS forms produce a wide variety of important information 
regarding the instructional activities used by the teachers, their duration, and the amount 
of on- and off-task behaviors of the students in the classroom. Data collectors worked 
with the school coordinator to identify three students for- these observations: one who 
was performing better than expected in the ESS program, one performing as expected, 
and one performing at a lower level than expected. Procedures were implemented so that 
neither the students nor teachers knew which student was being individually observed; 
this was managed either by using student photographs or unobtrusively pointing out 
specific students in the hallways or classrooms. The goal was to observe these target 
students during an ESS session, a language arts class, a mathematics class, and the 
subject for which they were referred to ESS (if other than language arts or mathematics). 

• QAIT assessment of classroom: This instrument is best described as a high-inference, 
simple coding, rating device. QAIT stands for Quality of Instruction, Appropriate Level 
of Instruction, Incentive, and Use of Time. Fitting on one 8 Vi" x 11 " sheet, it contains 
40 items grouped under the four major categories. Each item uses a Likert-type rating 
scale of 1 to 5 ( unlike this class to like this class). This instrument was to be completed 
at the end of each observation session. 

• Environmental resources: The Classroom Environment and Resources (CER) checklist 
is a low-inference, simple coding, sign system. Printed on the front of one 8 Vi" x 11" 
sheet, it contains 12 classroom attributes that are coded either as present or not present, 
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such as adequate lighting, use of multi-racial materials, posted assignments, etc. Next, 18 
classroom resource items, such as textbooks, computers, and worksheets are listed; 
observers indicate whether such resources are visible or not. If they are, observers 
indicate whether they are used during the observation. This instrument was to be 
completed at the end of each observation session. 

Data collection. Data collectors utilized these forms during the school site visits. The 
classroom observation segments were completed during the observation; QAIT and CER forms 
were completed as soon after the observation as feasible, given that scheduling sometimes 
required one observation to immediately follow another. Given the target of observing three 
students in at least three (possibly four) classrooms, the total number of expected completed 
SSOS forms per school could range from 9 to 12. Only ESS observations were completed 
during the summer visits, as regular school was not in session. 

Data analyses. After the fall/winter site visits were completed, AEL staff designed data 
entry templates using Remark scanning software. SSOS data were scanned by school; data files 
were then cleaned and exported to SPSS for statistical analyses. Although pretesting of the 
forms had been conducted prior to their use during the visits, problems were encountered in 
scanning the classroom observation segment of the SSOS. To ensure data validity, these data 
were entered by hand directly into SPSS. School files were merged into one master file before 
analyses began. A total of 193 student observations were completed during the fall/winter 
school visits and 20 were completed during the summer visits for a grand total of 213 
observations. 

Classroom observation data were averaged across the number of eight-minute intervals 
per each observation. Percentages of time for the classroom snapshots and target student 
activities were calculated for both the regular classroom and ESS sessions. Data were analyzed 
using the- 27 individual categories and-by grouping these into four main categories: teacher-led, 
student-led, management/organization, and off-task. Student engagement data and time spent by 
the target student in the four main instructional categories were also analyzed by ESS 
implementation patterns. 

QAIT data were analyzed by creating four main subscales composed of the 40 individual 
items. Descriptive statistics were used to describe results for both regular classrooms and ESS 
sessions. Further, independent t tests were conducted to determine if a statistically significant 
difference existed between the two types of classes for each of four categories: quality of 

instruction, appropriate level of instruction, incentive, and use of time. 

CER data were analyzed by calculating frequency percentages showing whether the 
classroom attributes were present in the regular classroom and ESS sessions. As well, frequency 
percentages were calculated to show whether various classroom resources were visible and used 
during the observations. 

Validity and reliability. These instruments were tested and utilized in the pilot test by 
Nesselrodt and Schaffer (2000a, 2000b). Prior to their use in this evaluation, AEL and KDE 
staff made final revisions to all three instruments and converted them to a scannable format. 
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Thus, these instruments possess face and content validity and have proven their utility in prior 
research. A high degree of inter-rater reliability was achieved among the data collectors, given 
that every participant passed at or above the 85% criterion of the SSOS coding assessment held 
at the conclusion of the training session. To assess the degree of internal consistency reliability, 
Cronbach alpha coefficients were computed for this administration of the SSOS and QAIT 
instruments, excluding demographic-type items; this procedure was not appropriate for the CER 
instrument, given its lack of variance in response options of either selected or not selected. For 
the SSOS instrument, this administration of the grouping strategy items resulted in a coefficient 
of .54; for the student engagement items, a .82. For the QAIT instrument, this administration of 
all items resulted in a coefficient of .94; by subscale, the coefficients were .91 for quality of 
instruction, .74 for appropriate level of instruction, .88 for incentives, and .80 for use of time. 



Other Instruments 

Instrumentation. Two other forms were utilized during the site visits. The School and 
Program Description Form was used in the pilot test and then refined by KDE and AEL staff for 
use in this evaluation. The form is machine scannable and fits on one sheet of 8 Vi" x 11" paper. 
It contains 15 items, 13 of which are demographic in nature, i.e., school characteristics, student 
enrollment, number of students and teachers involved in the ESS program, hours of operation, 
etc. The two open-ended items ask for a description of the major components of the ESS 
program and any unique characteristics of the school or community. See Appendix H for a copy 
of this form. 

The Innovation Component Configuration Map for Extended School Services (ICCM) 
was developed by the Kentucky Institute for Education Research (KIER, n.d.) in the mid-1990s. 
The ICCM is based on the Concems-Based Adoption Model (CBAM) originated and developed 
by Gene Hall, Shirley Hord, and others (Hall & Hord, 1987; Hord, Rutherford, Huling- Austin, & 
Hall, 1987). Basically, the ICCM is a map depicting the 15 major components of the ESS 
program in the three broad areas of student eligibility, school level program design, and district- 
wide ESS program planning. For each major component, there are three or four possible 
implementation variations (coded as 1 to 3 or 4, as appropriate), which were summed to create a 
total implementation score. The appropriate implementation levels for each component are 
determined through an analysis of all relevant data gathered during the site visit. When 
completed, a picture of the ESS component" configurations was established based on the 
implementation scores. When the ICCMs for all the site visits were completed, then a study of 
their various patterns of implementation was conducted; patterns were then used in conjunction 
with other variables to determine whether statistically significant differences occurred among the 
identified implementation patterns. See Appendix I for a copy of this instrument. 

Data collection. The School and Program Description Forms were to be completed and 
gathered during each of the 24 site visits. However, only 22 of these forms were returned to 
AEL for analysis. An ICCM form was completed by the data collectors for each school site visit 
and returned to AEL. 




33 



20 



Data analyses. A data entry template was developed for each instrument using Remark 
scanning software. After the instruments were scanned, the files were cleaned and then exported 
to SPSS for analysis. Descriptive statistics were generated for the School and Program 
Description Form items; the two open-ended items were qualitatively analyzed by common 
themes. Descriptive statistics were generated for the 15 ESS components on the ICCM. 
Further, AEL staff met to visually examine the school ICCMs to detect patterns of 
implementation. Based on this discussion, the only patterns emerging were based on the 
summed score for each school, which could range from 15 to 47 (the greater the score, the 
greater the degree of implementation). These patterns were used to correlate with a number of 
other school-level variables, including proficiency and accountability levels, attendance and 
retention rates, number of parent volunteer hours, average years of teaching experience, 
expenditures per student, and number of drug/weapon/assault incidents. These patterns were 
also used to determine statistically significant differences for student engagement and 
instructional activity categories from the SSOS, selected key items from ESS coordinator and 
teacher surveys, and the AEL CSIQ scales. 



Validity and reliability. The School and Program Description Form was developed, 
tested, and utilized in the pilot test by Nesselrodt and Schaffer (2000a, 2000b). Prior to its use 
for this evaluation, AEL and KDE staff made revisions and converted it to a scannable format. 
The ICCM is based on solid research endeavors such as the Concems-Based Adoption Model. 
Thus, these instruments possess face and content validity and have proven their utility in a prior 
administration. To assess the degree of internal consistency reliability, Cronbach alpha 
coefficients were computed for this administration of the ICCM; this procedure was not 
appropriate for the School and Program Description Form, given its strictly descriptive nature. 
This administration of all items in the ICCM instrument resulted in a coefficient of .82; by the 
three broad areas, the coefficients were .78 for student eligibility, .74 for school-level program 
design, and .44 for district-wide ESS program planning. 
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FINDINGS 



Findings are presented in two distinct sections. The first section presents comprehensive 
summaries of the individual data collection instruments so that all data are available for 
interpretation and use. Findings from the statewide ESS district and school coordinator surveys 
were presented earlier in a separate report (see Cowley & Meehan, 2001). The second section 
presents comprehensive summaries of data points that are directly aligned with the five broad 
evaluation topics and their related subquestions. 



Findings by Instrument 



AEL CSIQ 

The AEL Continuous School Improvement Questionnaire (AEL CSIQ) was administered 
to the 48 schools in the full sample. This 60-item instrument measures a faculty’s commitment 
to continuous improvement. Completed surveys were received from faculty members at 47 of 
the 48 schools, for a return rate of 98%. 

Each of the six scales contains 10 items, which respondents rated using a scale of 1 to 6 
(not present to present to a high degree ). These ratings were added together to form the scale 
■ scores, each of which has a possible range of 10 (1 x 10 items) to 60 (6 x 10 items). The names 
and definitions for the six scales are provided below. 

• Learning Culture: This scale reflects how well the culture of the school encourages 
learning by all— students, staff, and administration. It reflects the extent to which the 
school emphasizes learning rather than passive compliance, is a safe but exciting place to 
be, and encourages curiosity and exploration. It indicates the extent to which teachers 
have opportunities and encouragement to reflect on practice, work with others, and try 
new ways of teaching. 

• School/Family/Community Connections: This scale reflects the degree to which staff 

perceive that parents and community members are involved in and feel part of the school. 
This includes such activities as informing parents and community, forming meaningful 
partnerships, maintaining open communication, and honoring and respecting diverse 
points of view. 

• Shared Leadership: This scale reflects the extent to which staff view leadership as being 
shared — whether school administrators dominate decision making or there are 
mechanisms for involving teachers, students, and parents. It measures opportunities for 
leadership development and the extent of open, two-way communication. 
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• Shared Goals for Learning: This scale assesses the extent to which the school has clear, 
focused goals that are understood by all members of the school community. In addition, 
it reflects whether shared goals affect what is taught and how teachers teach, drive 
decisions about resources, focus on results for students, and are developed and "owned" 
by many rather than a few. 

• Purposeful Student Assessment: This scale reflects the extent to which respondents view 
student assessment data as meaningful; use data to guide instructional decisions; and 
believe data are communicated to the greater school community, including teachers, 
parents, students, and the general community. 

• Effective Teaching: This scale measures the extent to which teacher practice aligns with 
research on effective teaching. It assesses whether teachers actively engage students in a 
variety of learning tasks, pose questions that encourage reflection and higher-order 
thinking, expect students to think critically, and use teaching strategies designed to 
motivate students. 

Table 4 provides descriptive statistical summaries for the six scales for the total group 
and by building level. Because data were aggregated to the school level, all elementary scores 
were based on 23 schools, all middle school scores were based on 12 schools, and all high school 
scores were based on 12 schools. In general, elementary schools had higher mean scores and 
larger standard deviations, indicating greater dispersion within those scores than both the middle 
and high schools. Also, high schools had higher mean scores than the middle schools for five of 
the six scales. 

Figure 3 shows the mean scores for the schools within each of the three main building 
levels (elementary, middle, and high). Overall, the elementary school level had the highest mean 
scores for five of the six scales (the sixth was Shared Leadership, where the high school .level 
had the highest mean). The middle school level had the lowest mean scores for five of the six 
scales (the high school level had the lowest mean for Purposeful Student Assessment). The scale 
with the highest means across the three building levels was Shared Goals, followed closely by 
Purposeful Student Assessment and Effective Teaching. 

Data were also compared for the full group and at each building level by looking at the 
median split (50 th percentile) for the six scales. Table 5 provides the median scores for the total 
group and the elementary, middle, and high school building levels. At the full group level, 14 of 
the 47 schools (30%) scored above all six median, scale scores and 13 (28%) scored below all 
median scores. The median scores for the elementary schools were higher for each of the six 
scales, while scores for the middle and high schools were mixed. For the elementary schools, 6 
of the 23 schools (26%) scored above all six median scale scores and 8 (35%) scored below all 
median scores. The middle school scores were more equitably distributed, i.e., 4 of the 12 
schools (33%) scored above all six median scores and 4 (33%) scored below all median scores. 
For the high schools, 2 of the 12 schools (17%) scored above all six median scale scores and 4 
(33%) scored below all median scores. 
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for the Total Group and by Building Level 
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Scale Name 


Level 


Mean 


Standard Deviation 


Learning Culture 


Elementary 


49.61 


3.54 




Middle 


47.26 


2.43 




High 


48.10 


2.45 




Total 


48.63 


3.14 


School/Family/Community 


Elementary 


47.98 


4.82 


Connections 


Middle 


44.38 


3.60 




High 


46.81 


3.70 




Total 


46.76 


4.44 


Shared Leadership 


Elementary 


48.44 


5.74 




Middle 


47.36 


3.59 




High 


48.61 


2.86 




Total 


48.21 


4.59 


Shared Goals for Learning 


Elementary 


50.98 


4.03 




Middle 


48.53 


2.93 




High 


48.57 


2.48 




Total 


49.74 


3.58 


Purposeful Student 


Elementary 


50.97 


3.47 


Assessment 


Middle 


48.76 


2.37 




High 


47.86 


2.83 




Total 


49.61 


3.30 


Effective Teaching 


Elementary 


51.01 


3.02 




Middle 


48.42 


2.52 




High 


49.21 


2.43 




Total 


49.89 


2.93 
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Figure 3: AEL CSIQ Scale Mean Scores by Building Level 



Table 5: AEL CSIQ Scale Median Scores for the 
Total Group and by Building Level 



Scale Name 


Elementary 


Middle 


High 


Full Group 


Learning Culture 


49.52 


47.05 


48.67 


49.00 


School/Family/Community 

Connections 


49.04 


43.68 


47.47 


46.89 


Shared Leadership 


50.00 


47.71 


48.63 


49.13 


Shared Goals for Learning 


51.40 


49.27 


48.80 


50.06 


Purposeful Student 
Assessment 


51.44 


48.71 


47.83 


49.68 


Effective Teaching 


51.17 


48.10 


49.60 


49.93 
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ESS Student Questionnaire 

A total of 1,220 students participating in ESS in 24 selected sites responded to this 
survey. Eighteen site visits took place in the fall of 2001 and winter of 2002 and 6 visits took 
place in the summer of 2002. Due to missing data (skipped items), the number of respondents 
varied from item to item and was not reported; however, item completion in general ranged from 
1,134 to 1,220. 

When asked their school grade, 48% of ESS students responded that they were in high 
school (9th through 12th), 25% were in middle school (6th through 8th), and 28% were in 
elementary school (1st through 5th). The next item related to gender, and 52% of ESS students 
responded that they were male. 

Students then answered a series of items relating to their experiences with school and the 
ESS program by agreeing or disagreeing with the statements. Two thirds of ESS students (67%) 
agreed that they like school. When asked if they were a better student this year, 78% answered 
affirmatively. When asked if they attended school more often this year, 81% agreed. Eighty-six 
percent of ESS students said they asked for help in school when they needed it. When asked if 
they pay attention to their teachers, 87% agreed. Eighty-five percent of ESS students reported 
that their parent(s) asked them about school. 

When asked if they attended ESS this year, only 91% of the students agreed with the 
statement. Nine percent disagreed, indicating they did not understand the question or they did 
not know ESS was the extended school program name. Eighty-five percent of the students 
indicated the ESS program was helping them this year. Students were then asked if their ESS 
teacher lets them know how well they were doing; 75% agreed with this statement. Eighty-six 
percent of ESS students reported that they asked for help in ESS when they needed it. 

When asked what subjects they were working on in the ESS program, students could 
select any or all of the response options. Forty-eight percent indicated mathematics, 35% 
English, 32% reading, 24% other, 20% social studies, and 17% science. 

When asked what they liked best about the ESS program, 1,153 students (95%) 
responded with multiple comments. Almost one third (30%) of the comments were related to 
tutoring. One student replied, “I get the help that I need and I understand my work better.” 
Other comments were related to the extended time for homework (14%), making learning fun 
(12%), make-up work (9%), and individual instruction (9%). 

Finally, students were asked what would make ESS better; 1,055 (86%) students 
responded with multiple comments. Almost a fourth (19%) of the comments related to ESS 
scheduling. “More days per week and [make it] longer,” commented one student. Seventeen 
percent of students thought the program was good or nothing would make it better. Other 
comments were related to more games/fun activities (12%), more subjects offered (12%), and 
snacks (10%). 
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ESS Parent Questionnaire 

A total of 576 parents completed and returned the ESS Parent Questionnaire distributed 
in the fall semester of 2001 and the summer of 2002. Due to missing data (skipped items), the 
number of respondents varies from item to item and is not reported; however, item completion in 
general ranged from 531 to 576. Respondents were asked to identify the grade in which their 
ESS children were currently enrolled. Nearly a third (32%) reported that their children were 
enrolled in the elementary grades. Almost another third (30%) indicated that their children were 
in the middle or junior high school grades (5th grade through 8th grade), and more than a third 
(38%) that their children were in high school. 

Asked to identify all the individuals who had decided that their children should attend 
ESS activities, half (53%) indicated that their children’s teacher had, approximately a third 
(32%) that their children had, and somewhat more than a quarter (29%) that they (the responding 
parent) had. 

Most respondents (78%) reported that their children’s academic performance in school 
had become better or much better since their participation in ESS. A substantial percent (21%), 
however, indicated that their children’s performance had not changed, and 1% reported that it 
had worsened. 

Parents were asked to select from a list of options all those that described what their 
children had gained from ESS participation. More than half (58%) indicated that their children 
had acquired an improved understanding of the academic subject for which they had been 
referred to ESS. More than a third reported that their children were more confident (38%) or 
were passing the subject for which they had been referred (36%) as a result of ESS participation. 
According to 22% of respondents, ESS participation helped their children complete a grade 
successfully. Fewer reported that their children got along better at home (11%). Of those 
parents whose children were in a high school ESS session, 23% said that ESS will help their 
children graduate from high school. 

Asked how often they were notified of their children’s performance in ESS, a third (34%) 
of the parents replied that they were updated “once in a while.” Nearly another third (30%) 
reported that they were never notified. Seventeen percent were notified weekly, 12% monthly, 
and 7% daily. 

About a third (37%) of respondents indicated they were consulted about their children’s 
goals in the ESS program as needed throughout the school year. However, another third (32%) 
reported that they were never consulted. According to 19%, consultations about their children’s 
goals took place regularly throughout the year, and 12% indicated consultation had taken place 
only prior to the beginning of the school year. 

More than three fourths (81%) of responding parents reported that they understood the 
ESS program somewhat to fully; 15% reported that they understood it a little, and 4% indicated 
that they did not understand the program at all. 
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Parents were asked three open-ended questions on the survey. When asked to describe 
the best features of the ESS program, 461 of the 576 respondents replied, many with multiple 
comments. Nearly a fourth (21%) of the comments referred to the additional academic support 
ESS offered students. Fifteen percent noted that, due to ESS participation, students gained a 
more complete understanding of academic subjects. Fourteen percent of the comments indicated 
that one-on-one instruction is the best feature of ESS, followed by improved academic 
achievement of students (11%), and ESS allows students time to complete their homework, 
review content, or make up missed tests or other work (10%). Other important features included 
the flexible scheduling of ESS, the dedicated teachers, the provision of transportation to 
participating students, and the fact that ESS is free of charge. 

Of the 576 responding parents, 434 replied when asked to describe any problems with the 
ESS program. Nearly two thirds (64%) reported that there were no problems with the ESS 
program. On the other hand, 9% wrote that the lack of communication between ESS staff and 
families was a significant problem. As one parent put it, “I never hear how [my child] is doing.” 
Five percent each noted that ESS classes are not offered often enough and that ESS classes were 
not rigorous or focused enough. Said one parent, “They need to stop playing outside .... My 
child doesn’t need help with playing.” Other problems with the program that parents described 
included a lack of transportation for children attending ESS, insufficient numbers of teachers to 
support individualized instruction, and a limited number of subjects offered through ESS. 

Finally, parents of ESS students were asked whether their children would continue in 
ESS the following year, and for what reasons. Of the 512 parents who replied, half (51%) 
reported that their children would continue to attend. Slightly more than a third (34%) were 
uncertain, 8% reported that their children would not attend, and 7% replied that the question was 
not applicable to their children’s situation. 

Of those parents who reported that their children would continue to attend ESS, and who 
provided an explanation for their answer ( n = 257), more than a third (36%) reported this to be 
the case because ESS had been helpful thus far. “It is very helpful for these children that have a 
hard time,” wrote one such parent. Sixteen percent replied that their children would return to 
ESS because the program has enhanced their children’s academic performance. And for 12%, 
re-enrollment in ESS was warranted because their student continued to require additional 
assistance. Other reasons for re-enrollment included opportunities to keep abreast of academic 
work, the increased confidence students reportedly gain, and opportunities for individualized 
instruction. 

Overwhelmingly, most (73%) of those parents who reported that their children might or 
might not continue to attend ESS, and who provided an explanation for their reply (« = 162), 
wrote that their decision was contingent on whether their children appeared to need the 
additional help offered through the program. Nine percent wrote that they just were not certain 
about re-enrollment, and 7% reported that the decision was left up to their children. 

More than half (58%) of parents who replied that their children would not continue in 
ESS, and who offered an explanation of their reply, said this was the case because their children 
would not need assistance. As one such parent said it, “We believe she is up to speed.” Twenty- 
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one percent reported that their children’s grades had improved, whereas 14% wrote that their 
children’s grades had not improved. Other parents (7%) reported that their children would 
benefit more from the services of a private tutor. 

Seven percent of the 512 responding parents reported that they could not answer the 
query because it was not applicable to their children’s circumstances. Most of these reported that 
their children would be graduating from high school, and therefore re-enrollment in ESS was not 
an option. 



ESS Teacher Questionnaire 

A total of 225 ESS teachers in 24 selected sites responded to this survey. Eighteen site 
visits took place in the fall of 2001 and winter of 2002 and 6 visits took place in the summer of 
2002. Due to missing data (skipped items), the number of respondents varied from item to item 
and was not reported; however, item completion in general ranged from 190 to 225. 

The first three items of the ESS Teacher Questionnaire were demographic items: role, 
school level, and community. Ninety-two percent of the ESS teachers responded that their role 
in the school was a regular classroom teacher, 4% reported being ESS coordinators and regular 
classroom teachers, and 4% selected some other role. Forty-eight percent of the ESS teachers 
indicated teaching at the high school level, 21% at the middle or junior high level, and 31% at 
the elementary level. When asked to describe their community, 57% selected rural, 24% 
selected suburban, and 19% selected urban. ESS teachers. also reported an average number of 13 
students in their ESS classes. 

When asked how students were selected for ESS, the ESS teachers could select any or all 
of the response options. The most common response was teacher recommendation (87%), 
followed by parent request (56%), student request (45%), and standardized test scores (9%). Ten 
percent of the respondents selected other, most often citing counselor or administrator 
recommendation. 

When asked to indicate the most common reasons students received ESS services, 
respondents could select any or all of the response options. Eighty-four percent said to improve 
academic achievement, followed by in danger of failing (75%), to extend learning time (45%), to 
sustain present level of performance (43%), to improve self-esteem (30%), in danger of dropping 
out (11%), and other (10%). Respondents choosing other listed make up a credit, poor 
attendance, writing portfolio, more one-on-one instruction, computer time, make-up tests, and 
help with homework as reasons why students receive ESS. 

When asked what subjects were being taught in ESS, 81% of the teachers selected math, 
65% reading, 64% English, 54% social studies, and 53% science. Thirty-five percent selected 
other and most frequently listed subjects such as foreign language, writing portfolio, spelling, 
and computers. Respondents could select any or all of the response options. 
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When asked about staff development related to ESS, 53% of the teachers responded that 
they had received staff development. Of those, 98% said it was adequate. 

Teachers were asked about the frequency of consulting with the regular classroom 
teachers on the design of student instruction and target goals. Forty percent of the teachers 
indicated not applicable because they were the students’ regular classroom teachers. Twenty- 
nine percent said they consulted with the regular classroom teacher as needed throughout the 
school year, 22% selected regularly throughout the school year, 6% selected not at all, and 3% 
selected only prior to the start of school. 

When asked how frequently they consulted with the regular classroom teachers on 
student performance and progress, 45% of the teachers indicated not applicable because they 
were the student’s regular classroom teacher. Twenty percent each said they consulted with the 
regular classroom teacher at least once a month or at least once a week, 1 0% selected not at all, 
and 6% selected only at report card time. 

Teachers were then asked how frequently they consulted with parents on the design of 
individual student goals. More than half (58%) responded as needed throughout the school year 
and 25% said not at all. Fifteen percent of the respondents indicated regularly throughout the 
school year and 2% selected only prior to the start of school. When asked how frequently they 
consulted with parents on student performance and progress, 40% indicated at least once a 
month, 31% selected only at report card time, 16% said at least once a week, and 13% reported 
not at all. 

Seventy-three percent of the teachers responded that they monitored student performance 
and progress at least once a week. Eighteen percent of the respondents said at least once a 
month, and 4% each selected only at report card time or not at all. 

ESS teachers were asked how frequently they consulted with students on the design of 
their individual goals. Fifty-two percent of the teachers indicated as needed throughout the 
school year, 27% said regularly throughout the school year, 18% selected not at all, and 2% 
selected prior to the start of school. When asked how frequently they consulted with students on 
their performance and progress, 62% of the teachers reported at least once a week, 28% said at 
least once a month, 6% not at all, and 4% only at report card time. 

When asked to identify the most important ESS outcomes (respondents could select any 
or all of the response options), 95% of the ESS teachers selected enhanced academic 
achievement. About two-thirds of the teachers (65%) indicated increased motivation, 60% said 
increased self-esteem, and 26% selected improved attendance. 

When asked what forces have helped ESS to succeed (again respondents could select any 
or all of the response options), 64% responded excellent relationships among staff. Other forces 
selected were outstanding administration (principal/coordinator) (57%), clear support from 
parents or community (53%), additional financial support (43%), clear support or mandate from 
district or other political actions (40%), and excellent staff development and follow-up (21%). 
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Teachers were given a list of topics to indicate what problems or obstacles had been 
encountered in implementing ESS. Respondents could select any or all of the response options, 
as well as write in additional problems or obstacles. Almost half of the respondents indicated 
student transportation (40%). Other problems or obstacles identified were inadequate financial 
support (24%); opposition or demands from students (17%); inadequate preparation of teachers 
or other school staff (5%); problems with state or district regulations (4%); opposition or 
demands from key district, school, or other staff (4%); opposition or demands from parents or 
community (3%); and problematic relationships among school staff (1%). Thirteen percent 
chose other and listed problems or obstacles such as student attendance, lack of time, class size, 
lack of communication, and no problem. 

More than half (51%) of the teachers rated the effectiveness of the ESS program at their 
school as good. Forty-one percent selected excellent, 6% indicated fair, and 2% chose poor. 

When asked which option for disbursing ESS funds would be better for the students and 
schools, teachers were given two choices. Only 164 teachers responded to this question, which 
may be due to their lack of knowledge about how funds are disbursed. Overwhelmingly, 92% of 
the teachers selected the option to continue to provide as separate categorical funds allotted to 
districts. The teachers were then asked why they chose this option. Nearly one fourth (23%) of 
the respondents who chose this option reported that the funds would be kept separate. Thirteen 
percent wrote that the current method works well, 12% noted that the districts know the school 
needs and SEEK funds are not targeted for ESS, and 10% responded that they needed more 
information about the choices. 

Only 8% selected the option to provide ESS funds to districts through the SEEK formula. 
When asked why they chose this option, more than one fourth (27%) said the funding would be 
based on needs. Respondents also noted there would be money for transportation,, funding would 
be based on enrollment, and more information was needed about the choices (18% each). 

Of the 61 respondents who did not choose either option, there were 17 discrete comments 
provided about ESS funding options. Fifteen of the respondents (88%) reported that they didn’t 
know or they needed more information before making a choice. 

When asked about the major strengths of ESS at their schools, 193 teachers responded 
with multiple comments. Twenty-two percent of the comments dealt with the school staff. “We 
have an experienced staff which is dedicated to improving student performance and learning,” 
wrote one ESS teacher. Other strengths noted were improved academic achievement (8%), 
students (6%), and hours/times of ESS (6%). 

When asked about the biggest challenges faced by ESS at their schools, 180 teachers 
responded with multiple comments. One fourth of the comments concerned students. One 
teacher replied, “The biggest challenge is convincing students to use it for actual tutoring.” 
Eighteen percent reported transportation was a challenge and 15% reported funding issues. 

Of the 225 ESS teachers surveyed, 136 responded with multiple replies when asked what 
recommendations they would make to improve ESS. Nearly a fourth (22%) of the comments 
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concerned funding. “Adequately fund the program in order to make a difference,” noted one 
ESS teacher. Other comments were related to ESS class size (12%), scheduling (8%), and 
transportation (7%). 

Finally, ESS teachers were asked what else we should know about ESS. More than half 
(54%) provided positive comments. “It is an excellent program that increases student self- 
esteem and improves their academic performance,” reported one ESS teacher. “Involvement 
with students in the ESS situation is rewarding because I can have more one-on-one contact with 
students who want help,” wrote one teacher. Another teacher commented, “We are extremely 
proud of the program.” The remaining comments related to students (14%), nothing or not 
applicable (8%), and funding (4%). 



Non-ESS Teacher Questionnaire 

A total of 297 non-ESS teachers responded to this survey. Due to missing data (skipped 
items), the number of respondents changed from item to item and was not reported; however, 
item completion in general ranged from 281 to 297. 

The questionnaire included three demographic items: role, school level, and community. 
Ninety-two percent of the respondents indicated that they were classroom teachers, 1% indicated 
principal or assistant principal, and 6% reported other. Fifty-six percent of the respondents said 
that they worked in a high school, 23% indicated a middle school, and 21% said elementary 
school. Thirty-eight percent of the teachers said that their school was located in a rural 
community, 37% indicated a suburban community, and 26% indicated an urban community. 

Teachers were then asked how many of their students were enrolled in ESS; percentages 
ranged from 0 to 99%. Respondents indicated that, on average, eight students per class were 
enrolled in ESS. Twenty-three percent indicated that none of their students were enrolled and 
1% indicated that 99% of their students were enrolled. 

When asked how most of their students were selected for ESS, respondents could select 
any or all of the response options. More than three fourths (82%) indicated teacher 
recommendation, followed by parent request (44%), student request (33%), standardized test 
scores (9%), and other (8%), with responses such as counselor, grades, reading test scores. 

Teachers were asked about the most common reasons their students received ESS; nearly 
three fourths (73%) each selected students in danger of failing or to improve academic 
achievement. Twenty-six percent selected to extend learning time, 18% to sustain present level 
of performance, 17% to improve self-esteem, and 5% indicated students that were in danger of 
dropping out. 

When asked what subjects their students were receiving instruction on in the ESS 
program, more than half (59%) reported math, followed by reading and English (47% each), 
science (33%), social studies (28%), and other (14%). For the respondents who indicated other, 
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the most common answer reported was writing (respondents could select any or all of the 
response options). 

Thirty-six percent of the respondents indicated that they received staff development 
related to ESS. Of those, nearly all (98%) reported that the staff development was adequate. 

Teachers were then asked how frequently they consulted with ESS teachers on the design 
of student instruction and target goals. More than half (51%) reported as needed throughout the 
year, 31% said not at all, and 15% indicated regularly throughout the school year. 

When asked how frequently they consulted with ESS teachers on student performance 
and progress, 42% reported not at all, 30% indicated at least once a month, and 14% each said 
only at report card time or at least once a week. 

When asked about the most important ESS outcomes for their students, teachers could 
select any or all of the response options. The majority of the respondents (89%) indicated 
enhanced academic achievement, followed by increased motivation (38%), increased self-esteem 
(28%), improved attendance (13%), and other (5%) with comments such as completion of 
assignments, graduate with classmates, and improve school’s CATs scores. 

When asked what forces helped ESS to succeed at their schools, 52% reported excellent 
relationships among staff, 44% said outstanding administration, 39% indicated additional 
financial support, 38% selected clear support from parents or community, 31% reported clear 
support or mandate from district or other political actions, and 29% said excellent staff 
development and follow-up. Again, respondents could select any or all of the response options. 

The next question asked teachers to identify problems or obstacles that had been 
encountered in implementing ESS at their school. Respondents could select any or all of the 
response options. A third of the teachers (33%) indicated student transportation; 32% said 
inadequate financial support; 21% reported opposition or demands from students; 8% each said 
problems with state or district regulations and opposition or demands from parents or 
community; 7% reported inadequate preparation of teachers or other staff; 2% indicated 
opposition or demands from key district, school, or other staff; and 1% said problematic 
relationships among school staff. Eleven percent reported other, with common responses being 
available teachers, funding, getting the students to recognize the great benefits of ESS, and lack 
of student motivation to become involved. 

Teachers were then asked how they would rate the overall effectiveness of ESS at their 
school. More than half (54%) reported good, 34% said excellent, 11% indicated fair, and 1% 
reported poor. 

When asked which option for disbursing ESS funds would be better for the students and 
school, only 174 teachers responded. Of those, nearly all (91%) said continue to provide as 
separate categorical funds allotted to districts. Respondents were then asked why they chose this 
option. Seventeen percent indicated that it is working well so far with no need to change it, 10% 
indicated that “Allotting separate categorical funds allows us to specifically utilize this funding 
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for better uses to benefit our program” and “Districts can best meet their needs because of the 
size of the smaller districts and the needs can be assessed on the local level.” Nine percent said 
they did not know enough about the SEEK formula to select that choice. Other comments 

included “To make sure that it isn’t used for something else and then ESS would be under 
funded or eliminated” and “Funds would be earmarked for ESS at individual schools.” 

Only 9% indicated the funds should be provided to districts through the SEEK formula. 
When asked why they chose this option, 30% indicated to improve student achievement, and 
20% reported more flexible use of the money. Other comments included explanations such as 
the SEEK formula could provide transportation for students, money goes directly to schools 
through the district, and more evenly distribute those funds on student population and need. 

Of the 123 respondents who did not select either option, 1 1 provided a written comment. 
Of those, 64% reported that they were not knowledgeable enough either way to make a choice 
and 18% said they were unfamiliar with the SEEK formula. 

When asked about the major strengths of ESS at their school, 229 teachers responded 
with multiple comments. Thirty-six percent of the comments indicated that having excellent 
staff was a major strength, 34% said after-school help, 16% said adequate transportation, 12% 
reported coordination between teachers/coordinators, 7% responded teacher/student ratio, 6% 
reported flexibility, and 5% said student improvement. Four percent indicated other, with 
comments such as removing obstacles to learning and targeting students as soon as possible. 
One teacher commented, “We have teachers willing to work overtime with little pay and with 
students that are tired and wanting to go home.” 

When asked to identify the biggest challenges faced by ESS at their schools, 211 teachers 
responded with multiple comments. Almost one fourth (18%) of the comments referred to 
funding as the biggest concern. Fifteen percent of the comments related to student attendance, 
12% to transportation by the school, 11% to teacher/student ratio, 7% to student motivation, 5% 
to staff communication; 4% each reported the amount of paperwork, student needs, and not 
enough knowledge about the program to comment. Two percent reported other, with comments 
such as “Some after school commitments interfere, such as jobs” and “All ESS happens on the 
same day so students are limited to attending 1 or 2 subjects.” One teacher noted, “Some students 
who need the service refuse it.” 

When asked for recommendations to improve ESS, 158 teachers provided multiple 
■comments. Almost one third (29%) of the comments referred to more funding. “More funding 
would allow students to be served for a longer period of time,” commented one teacher. “More 
funding may also allow us to offer the program in the mornings” stated another. Eleven percent 
indicated more teachers are needed and 8% each reported scheduling, staff, or other. For those 
responding with other, comments included pleased with the program, enrichment programs 
added, make the program accessible to all students, and ability to change students in ESS 
throughout the school year. Seven percent each reported transportation or no recommendations, 
5% each indicated attendance and reduced paperwork, and 4% each indicated communication, 
tutoring, or more instruction. 
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Finally, the last question asked teachers what else we should know about ESS. Sixty- five 
teachers responded with multiple comments. Nearly half (40%) of the comments indicated that 
teachers supported the program or were pleased with the program. “It is a good program that 
helps kids,” noted one teacher. “I think it is wonderful to allow students another opportunity to 
achieve in school,” wrote another teacher. Eighteen percent of the comments indicated student 
needs and 16% indicated respondents had nothing else to say about ESS. The remaining 
comments related to staff (12%), the need for additional funding (11%), and other (9%) with 
comments related to the need for more information and scheduling of ESS. 



District Coordinator Interviews 

Three of the district coordinators had two of the site visit schools in their districts. 
Fifteen district coordinator interviews were conducted during the fall/winter site visits. No 
additional district interviews were done during the summer visits to six of the same schools 
because these individuals were previously interviewed. Interviews were conducted at the school 
site, by telephone, or at the district office. Brief summaries of key questions are summarized 
below, with their most frequent responses.* 

District coordinators were asked how eligibility is determined. One fourth of the 
coordinators said it was determined by district guidelines, 20% indicated it was a combination of 
factors including reviewing a list of failing students, test analysis, and selecting the worst first, 
and 18% said teacher referral. 

Coordinators were then asked if there was a formal method of referral. Forty percent 
indicated that teachers initiate the referrals and parents must consent. Ten percent said 
communities were made aware through television, radio, or newspaper ads; 10% said there was 
no formal referral method or they didn’t know; and another 10% of the comments were 
miscellaneous, such as computer database and method has to be approved. 

District coordinators were asked to describe the services that were offered in their 
districts. Forty-three percent of the comments were regarding the days and times ESS was 
offered. Other services offered were budget monitoring (12%), and administrative services 
(9%). 

When asked about the main problems the program was intended to solve, 45% responded 
to help students succeed. . Twenty-eight percent of the comments were miscellaneous and 
included make up lost credits, raise attendance, and higher retention rates, and 17% said it is 
intended to solve problems in the content specific areas. 



*See the supplemental document to this report, “Interview Questions and Response Categories for the 
Comprehensive Evaluation of the Kentucky Extended School Services Program ,” for tables containing all analyzed 
interview questions and categorized comments for each of the five interviewee groups (district coordinators, school 
coordinators, ESS teachers, ESS students, and parents of ESS students). 
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School Coordinator Interviews 

Twenty-three school coordinator interviews were conducted across the 24 sites. One 
interview was not completed due to an accident involving the coordinator. All interviews were 
conducted at each school site. Brief summaries of key questions are provided below, with their 
most frequent responses. 

School coordinators were asked how eligibility for ESS was determined and if there was 
a formal method of referral. Almost one third (31%) indicated students were referred by their 
teachers, 20% said parents refer students, and 10% indicated students self-refer. Twenty-two 
percent responded with miscellaneous comments such as students are selected at the beginning 
of the year, chosen by the guidance counselor, or those with the most need are selected. The 
formal method of referral most often mentioned was a referral sheet completed by teachers or 
parents (51%). 

When asked to describe the services provided in ESS, 20% of the school coordinators 
indicated ESS focused on the content areas. Sixteen percent said ESS provided remediation and 
study skills, and 9% said tutorial services. Twenty percent of the comments were miscellaneous, 
such as improve students’ self-esteem, mentoring, and transportation. 

School coordinators were then asked about the main problems the program was intended 
to solve. Almost half (43%) said the main goal is to help students succeed. Twenty-three 
percent indicated assistance in content areas, and 16% responded with miscellaneous comments, 
such as flexible for all students and home visits for sick students. 



ESS Teacher Interviews 

Ninety-eight ESS teachers were interviewed across the 24 site visits. All teacher 
interviews were conducted individually at the schools. Brief summaries of key questions are 
provided below, with their most frequent responses. 

Teachers were asked how students were selected for participation in the ESS program. 
Twenty-four percent indicated teacher recommendation or referral, 18% said parent request, 16% 
were selected after being identified as having problems or struggling, and 14% were self- 
referred. 

Teachers were then asked to describe the key elements of the ESS curriculum at their 
schools. Eighteen percent each indicated that everything was the same as the regular classroom, 
reading in content areas, or math were the key elements of the curriculum. Twelve percent 
responded with miscellaneous comments, such as homework, study skills, and book club. 

When asked about the main problems the program was intended to solve, 23% of 
teachers responded the goal is to help struggling learners. Thirteen percent each noted to 
increase confidence and motivation, improve student performance, and prevent failure or 
dropout. 
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ESS Student Interviews 

Most student interviews were conducted individually at the high school level and in 
groups of two at the middle/junior high school and elementary school levels. There were 109 
students interviewed across the 24 site visits. Brief summaries of key questions are provided 
below, with their most frequent responses. 

Students were asked why they participated in ESS and if they chose to participate on their 
own or if their teachers or parents recommended it. Nearly a third of students said they 
volunteered to participate in ESS to receive more help and to improve their grades. Eighteen 
percent indicated their parents wanted them to participate, and 1 7% said their teachers 
recommended it. 

Students were then asked what they learned about in ESS. Sixteen percent responded 
with various activities, such as listening to tapes, flash cards, puzzles, worksheets, etc. Students 
specified English or language arts (14%), reading (13%), and math (9%) as the main subjects 
they learned about in ESS. 



ESS Parent Interviews 

Forty-nine parents were interviewed across the 24 site visits. Parent interviews were 
conducted by telephone or individually at the school. Brief summaries of key questions are 
provided below, with their most frequent responses. 

Parents were asked why their children were participating in ESS. Thirty percent of the 
comments were miscellaneous, such as to catch up on schoolwork and to make up tests. About 
one fourth (22%) indicated their children participated due to a need for extra help in a specific 
subject, and 16% said their children needed additional help. 

When asked how their children were doing since participating in ESS, more than half 
(54%) indicated their children’s grades had improved. Seventeen percent said the process was 
helping, and 12% of the comments were miscellaneous, such as child’s attitude could be better 
and the program just started. 

Parents were then asked what they thought their children had gained from the ESS 
program. Nearly one fourth (23%) said their children get more attention, 15% indicated better 
grades were a result, and 14% said improved self-esteem. 
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SSOS Classroom Observations 

A total of 24 on-site visits took place in 18 selected Kentucky schools. Eighteen of the 
visits took place during the fall and winter of 2001 to schools operating ESS programs after 
school; the remainder took place during the summer of 2002 to 6 of the 18 schools that were 
operating ESS programs during a summer session. Six of the 18 schools were elementary, 6 
were middle, and 6 were high schools (33% each). 

During the 24 visits, a total of 213 classroom observations took place: 193 (91%) during 
the fall/winter visits and 20 (9%) during the summer visits. These observations focused on both 
entire classrooms of students and purposively selected ESS students within each classroom. Of 
those 213 observations, 137 (64%) took place in the regular classroom and 76 (36%) in an ESS 
session. The goal was to observe selected students in both their regular classroom environments 
and their ESS sessions. This was not always achieved, because some students were selected and 
observed during the regular school day, but then did not stay after school for ESS. 

Observed students were grouped into three categories: performing better than expected 
in the ESS sessions (coded as 1), performing about as expected in the ESS sessions (coded as 2), 
and performing less well than expected in the ESS sessions (coded as 3). Approximately a third 
of the students were in each of the three categories: better than expected (67, 32%), as expected 
(73, 34%), and less than expected (73, 34%). 

Within the ESS sessions, the number of adults per classroom ranged from 1 to 5, with an 
average of 1 (standard deviation of 0.61). The number of . students in the ESS sessions ranged 
from 2 to 31, with an average number of 1 1 students (standard deviation of 6.54). In the regular 
classrooms, the number of adults ranged from 1 to 4, with an average of 1 (standard deviation of 
0.58). The number of students in the regular classrooms ranged from 12 to 41, with an average 
number of 22 students (standard deviation of 5.08). 

A variety of subjects were observed within both the ESS and regular classroom 
observations, as shown in Table 6. The biggest differences seemed to be in reading and science. 
Fourteen percent of the regular observations focused on reading, compared to 4% of the ESS 
sessions; 1% of the regular observations focused on science, compared to 12% of the ESS 
sessions. Math was fairly comparable for both sessions (34% and 40%). 

The classroom observation coding tool contained both a “class snapshot” that looked at 
student engagement and grouping configurations every eight minutes across the entire classroom 
and “ongoing activities of target student” that focused on the student being observed for seven 
consecutive minutes. During those seven minutes, 27 different activities could be coded in time 
segments of 1 to 7 minutes. Each classroom observation could last a maximum of 56 minutes, 
which did not include the time necessary to code in relevant information on the cover page (such 
as number of students, number of adults, subject, date, type of class, etc.) or the accompanying 
QAIT and CER instruments. 
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Table 6: Number and Percent of Classes Observed by Subject and Class Type 



ESS Classrooms 


Regular Classrooms 


Subject 


Number 


Percent 


Subject 


Number 


Percent 


English 


12 


16% 


English 


15 


11% 


History 


1 


1% 


History 


2 


2% 


Language Arts 


13 


17% 


Language Arts 


28 


21% 


Mathematics 


26 


34% 


Mathematics 


54 


40% 


Reading 


11 


14% 


Reading 


6 


4% 


Science 


1 


1% 


Science 


17 


12% 


Social Studies 


3 


4% 


Social Studies 


9 


7% 


Writing 


1 


1% 


Writing 


0 


— 


Other 


8 


10% 


Other 


5 


4% 


Total 


76 


100% 


Total 


136 


100% 



Student engagement. The four categories within student engagement included number 
of students on task, number of students off task, number of students out of the room, and number 
of students waiting. The average number of students on task in the ESS sessions was 10 
(standard deviation of 6); in the regular classroom, the average number on task was 19 (standard 
deviation of 5). For the remaining three categories, averages were approximately 1 for both 
groups (with larger standard deviations for the regular classroom). Figure 4 presents the 
percentages of students in each of the four categories by type of class (ESS or regular) and shows 
very similar results for the two groups. Students on task was clearly the most prevalent view 
during the “snapshots,” with 87% of the ESS students and 88% of the regular students falling 
within this category. Off-task was a distant second, with 6% of ESS students and 5% of regular 
students. The remaining two categories were negligible, with only 3 or 4% percent each. 



On Task 



Off Task 



Out of Room 



Waiting 




Percent of Students 



Figure 4: Observation Data: Percent of Student Engagement by Class Type 
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Groups and activities. Four groupings were possible within this section of the 
observation instrument. One grouping was based on the teacher, one on an aide, and two were 
for remaining discrete student groupings (for this analysis, the two student groupings were 
combined into one student group). Within each group, four tasks were possible: interaction, 
working alone, management, and social.* Data analyses revealed a discrepancy in the working 
alone task for aides, in that an inordinately high percent of students were involved (10% to 20% 
per type of class), which may have been due to either coder drift or data entry errors. Therefore, 
this aide task classification is excluded from further analyses. 

Within the teacher grouping of ESS sessions, they were involved with the most students 
when interacting (student mean of 6, standard deviation of 6) or managing (student mean of 5, 
standard deviation of 6). Results for aides were similar, with interacting (student mean of 3, 
standard deviation of 2) and managing (student mean of 3, standard deviation of 4). Students 
differed, with most involvement with other students when interacting (student mean of 6, 
standard deviation of 5); students also were working alone (student mean of 8, standard deviation 
of 6). 



Within the regular classrooms, patterns were similar, though of course with higher mean 
numbers of students, given the larger class sizes. For teachers, interaction (student mean of 17, 
standard deviation of 7) and management (student mean of 15, standard deviation of 9) involved 
the highest percentages of students; for aides, interaction (student mean of 6, standard deviation 
of 7); for students, interaction (student mean of 15, standard deviation of 6) and working alone 
(student mean of 16, standard deviation of 6). 

The next three figures present the percentages of students involved in each of four tasks 
across the three groups of teachers, aides, and students. These percentages were based on the 
number of observations involving each type of task, which varied within each of the three 
groups. As a result, the percentages within each graph for a particular class type (i.e., ESS or 
regular) should not and do not sum to 100. 

Figure 5 presents the percentages of students by type of class (ESS or regular) for each of 
the four teacher groups of involvement, working alone, management, and social. As expected, 
highest student percentages appeared in interaction and management, with higher percentages for 
the regular classes in both categories. ESS teacher interaction was with 53% of the students, 
compared to 77% of the regular students; ESS teacher management involved 48% of the 
students, compared to 67% of the regular students. ESS teacher socialization ranged from 1% of 
the students in the ESS classroom to 25% in the regular classroom. 

Figure 6 presents the percentages of students for two of the three aide groups (no aide 
socialization was coded and aide working alone was excluded). Highest student percentages 
appeared in interaction, with 27% of the ESS students and 21% of the regular students. Aide 
management involved 19% of the ESS students and 3% of the regular students. 



interaction includes content-related interactive activities; work alone indicates working on content individually; 
management/direction indicates monitoring, managing, or teaching non-content-related procedures; social indicates 
uninvolvement in content-related or managing work. 
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Figure 5: Observation Data: Percent of Students in Teacher Groups by Class Type 
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Figure 6: Observation Data: Percent of Students in Aide Groups by Class Type 
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Figure 7 presents the percentages of students for each of the three student groups (no 
student management was coded). Working alone was the most frequent student grouping, 
including 71% of the ESS students and 75% of the regular students. This was followed by 
student interaction, involving 46% of the ESS students and 69% of the regular students. Student 
socialization involved 25% of the ESS students and 20% of the regular students. 




Percent of Students 

Figure 7: Observation Data: Percent of Students in Student Groups by Class Type 



Ongoing activities of target student. This section of the classroom observation 
instrument included two main components, along with a small column for writing notes and/or 
comments. The first component was to note which of the 27 different activities the observed 
student was engaged in during the observation. The second component was to indicate how 
many minutes were spent in a particular activity. The smallest time increment was one minute; 
the largest was eight minutes (each page of an observation was for a maximum of eight minutes, 
with a maximum of seven pages per observation, or 56 minutes of observed time). 

To provide an overall look at the time spent per activity, Figure 8 presents the percentage 
of time spent on each of the 27 activities by the type of class (i.e., ESS or regular). Individual 
student seatwork was used 26% of the time in the ESS sessions and recitation/discussion was 
used at least 10% of the time. During the regular classroom, recitation/discussion, teacher 
presentation of material, and individual student seatwork were each used at least 15% of the 
time. 
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Figure 8: Observation Data: Percent of Target Students’ Time 
Spent on All Activities by Class Type 



The 27 activities were then grouped into four main categories: teacher-led, 

management/organization, student-led, and off task. See Table 7 for a listing of the 27 activities 
and their classification into the four main categories. Figure 9 shows that for the ESS sessions, 
nearly half (48%) of the time was spent on student-led activities. In Comparison, more than half 
(51%) of the time in the regular classrooms was spent in teacher-led activities. Conversely, the 
regular classrooms devoted only 28% of the time to student-led activities, and the ESS sessions 
devoted only 32% to teacher-led activities. Nearly equal amounts of time were spent in off-task 
activities for both the ESS and regular classrooms (14% and 15%, respectively). Only 6% of the 
time in each type of class was spent on management/organization activities. 
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Table 7: Classroom Observation Activities by Main Categories 



Main Categories 


Activities 


Teacher-Led 


Teacher presentation of content 

Recitation or discussion 

Directions for assignments 

Small-group instruction 

Tests 

Checking 

Praising class 


Management/Organization 


Procedural or behavioral presentation 
Administrative routines 
Transitions 
Monitoring 


Student-Led 


Individual seatwork 

Individual seatwork at computer 

Pairs or group seatwork 

Pairs or group seatwork at computer 

Sustained writing or composition 

Sustained reading 

Hands-on learning 

Independent inquiry or research 

Student-initiated questions 


Off Task 


Teacher nonacademic activity 

Waiting time 

Discipline 

Student nonacademic activity 
Not occupied 
Off task 
Out of room 



Given that the amount of time spent in each main category (or even a specific activity) 
could theoretically range from 1 to 56 minutes, the average number of minutes spent in each of 
these four main groups is not quite as useful as the overall percentages of time. However, these 
data are provided in Table 8, along with the number of observations and the standard deviations. 
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Figure 9: Observation Data: Percent of Time in Main Activity Groups by Class Type 



Table 8: Observation Data: Descriptive Statistics for 
Main Activity Categories by Class Type 



Type of Class and Main Categories 


Number 


Mean 


Std. Dev. 


ESS Teacher-Led 


55 


19.64 


14.80 


Management/Organization 


39 


4.69 


3.64 


Student-Led 


62 


25.55 


14.53 


OffTask 


57 


8.33 


7.75 


Regular Teacher-Led 


131 


24.13 


14.15 


Management/Organization 


100 


3.66 


3.38 


Student-Led 


104 


16.67 


12.96 


OffTask 


108 


8.88 


8.19 



To further pinpoint exact differences within each of these four broad main categories, 
Figures 10 through 13 present percentages of time spent on individual activities within the 
teacher-led, management/organization, student-led, and off-task categories. 
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For example, the 51% of time spent in the regular classroom on teacher-led activities can 
be identified in Figure 10 as due mainly to teacher presentation of content and recitation/ 
discussion. In comparison, the 32% of time spent on teacher-led activities in the ESS sessions is 
more broadly distributed across teacher presentation of content, recitation/discussion, small- 
group instruction, and testing. 



Tchr. presen. 
Recit./disc. 
Directions 
Sm.-Grp. instr. 
Tests 
Checking 
Praising class 




1'.! ESS 

Regular 



0 5 10 15 20 

Percent of Time Spent on Activity 



Figure 10: Observation Data: Percent of Time Spent in 
Teacher-Led Activities by Class Type 



Management/organization activities are similar for both ESS and regular classrooms, 
with only minimal differences in transitions and monitoring (see Figure 11). 

The ESS sessions spent nearly half of the time (48%) on student-led activities, compared 
to only 28% of the regular classroom time. Figure 12 shows that most of that time was spent on 
individual student seatwork in both types of class, though more so for the ESS sessions (26% 
compared to 15%). 

For the main category of off task, which includes a variety of activities classified as being 
off task, both types of classrooms devoted approximately 15% of the time to this behavior. 
Review of Figure 13 shows fairly comparable specific activities, most especially actual off-task 
behavior (5% and 6% for ESS and regular classrooms), students not being occupied, and students 
being out of the room. 
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Figure 1 1 : Observation Data: Percent of Time Spent in 
Management/Organization Activities by Class Type 
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Figure 12: Observation Data: Percent of Time Spent in 
Student-Led Activities by Class Type 
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Figure 13: Observation Data: Percent of Time Spent in 
Off Task Activities by Class Type 



QAIT Assessment of Classroom 

The QAIT Assessment of Classroom observation instrument measured four features of 
the classroom: (1) quality of instruction, (2) appropriate level of instruction*, (3) incentive, and 
(4) use of time. Forty specific features were rated on a 1 to 5 ( unlike this class to like this class). 
Across the 18 schools, 213 classrooms were observed and rated with the QAIT, including 193 
observations during the fall and winter site visits and 20 ESS classroom observations during the 
summer site visits. Results are presented below, organized by the four main classroom features, 
and disaggregated by type of class to make apparent any potential differences between ESS 
classrooms and regular classrooms. Seventy-six ESS classroom observations and 137 regular 
classroom observations were included. 



*One item from this section about homogeneity of ability was deleted from the analysis as it did not fit well with the 
other items and was not under the teacher’s control. 
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Quality of Instruction. Teachers in regular classrooms organized lessons in ways that 
made sense to students to a greater extent than did teachers in ESS classrooms, and more often 
stated lesson objectives orally or in writing. They were also more likely to remind students of 
previously learned material. Teachers in both types of classrooms exhibited enthusiasm to a 
moderately high or moderate degree, and some teachers exhibited a sense of humor. Teachers in 
both types of classrooms often used an appropriate pace to cover content and provided 
immediate and corrective feedback. See Table 9 for more details. 

Table 9: Observation Data: Descriptive Statistics 
for Quality of Instruction by Class Type 



Indicators 


ESS Classroom 


Regular Classroom 


N 


Mean 


SD 


N 


Mean 


SD 


1 . Lessons make sense to students. The 
teacher: 














Organizes information in an orderly way. 


72 


3.39 


1.57 


133 


3.91 


1.22 


Notes transitions to new topics. 


70 


2.51 


1.67 


134 




3.40 




1.41 


Uses many vivid images and examples. 


73 


2.48 


1.56 


131 


2.98 


1.52 


Frequently restates essential principles. 


72 


3.03 


1.64 


132 


3.70 


1.43 


2. Lessons relate to students’ background. 
The teacher: 














Uses devices such as advanced 
organizers. 


73 


2.22 


1.53 


131 


2.70 


1.60 


Reminds students of previously learned 
materials. 


73 


3.25 


1.49 


133 


3.74 


1.27 


3. The teacher exhibits enthusiasm. 


74 


3.72 


1.34 


134 


3.72 


1.26 


4. The teacher shows a sense of humor. 


73 


3.04 


1.50 


135 


3.13 


1.51 


5. Lesson objectives are clearly specified. 
The teacher: 














States lesson objectives orally or in 
writing. 


73 


2.63 


1.58 


133 


3.56 


1.40 


Conducts formal and/or informal 
assessment. 


72 


3.37 


1.58 


134 


3.64 


1.38 


Provides immediate and corrective 
feedback. 


73 


3.75 


1.58 


134 




3.78 

- 


1.31 


6. Teachers use an appropriate pace to cover 
content. 


73 


3.71 


1.48 


134 


3.84 


1.28 
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Appropriate Level of Instruction. Teachers in the ESS classrooms seemed to make 
more use of instructional strategies matching students’ abilities than did teachers in the regular 
classrooms. These ESS classrooms also contained higher levels of individual instruction. In 
neither type of classroom were in-class ability groups or cooperative learning arrangements 
commonly observed. See Table 10 for more details. 

Table 10: Observation Data: Descriptive Statistics 
for Appropriate Level of Instruction by Class Type 



Indicators 


ESS Classroom 


Regular Classroom 


N 


Mean 


SD 


N 


Mean 


SD 


7. Instructional strategies match students’ 
abilities. The teacher: 














Accommodates students’ levels of prior 
knowledge. 


74 


3.74 


1.35 


135 


3.21 


1.41 


Accommodates students’ different 
learning rates. 


74 


3.64 


1.48 


135 


2.84 


1.38 


8. Grouping strategies enable students to 
work together or alone. The teacher: 














Uses in-class ability grouping. 


74 


1.58 


1.21 


132 


1.32 

— 


0.93 

— 


Uses cooperative learning arrangements. 


74 


1.73 


1.34 


134 


1.56 


1.22 


Bases individual instruction on mastery of 
skills and/or concepts. 


74 


2.88 


1.55 


134 


2.01 

— 


1.37 

— 


Uses individualized instruction. 


74 


3.43 


1.67 


135 


1.87 


1.28 



Incentive. Teachers in neither type of classroom appeared to have used surprising 
demonstrations to arouse student curiosity. In these classrooms, students were more likely to be 
allowed to discover information and be provided with intrinsically interesting material than the 
other listed methods for arousing curiosity. Teachers used praise and feedback, accountability, 
waiting for responses, guiding partial responses, and communicating high expectations to a 
moderate degree, though with the large standard deviations these seemed to differ a great deal 
among classrooms of each type (ESS and regular). Praise, tokens, and group contingencies were 
not often witnessed as methods for extrinsic behavioral incentives. Teachers did provide 
instruction so that student efforts would lead to success in both types of classrooms, though 
slightly more so in ESS than in the regular classrooms. See Table 1 1 for more details. 

Use of Time. Time needed for instruction seemed to be allocated within both ESS and 
regular classrooms. The ESS classrooms appeared to have slightly higher engagement rates in 
the form of students attending to lessons and the teacher using effective management. See Table 
12 for more details. 
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Table 1 1 : Observation Data: Descriptive Statistics 
for Incentives by Class Type 
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Indicators 


ESS Classroom 


Regular Classroom 


N 


Mean 


SD 


N 


Mean 


SD 


9. The teacher arouses students’ curiosity by: 














Presenting surprising demonstrations. 


72 


1.68 


1.25 


135 


1.76 


1.16 


Relating topics to students’ lives. 


72 


2.21 


1.50 


136 


2.51 


1.57 


Allowing students to discover 
information. 


72 


2.78 


1.61 


134 


2.94 


1.52 


Presenting intrinsically interesting 
material. 


71 


2.61 


1.59 


134 


2.83 


1.42 


10. The teacher uses extrinsic academic 
incentives such as: 














Praise and feedback. 


72 


2.96 


1.48 


134 


3.04 


1.38 


Accountability. 


73 


2.99 


1.52 


133 


3.16 


1.40 


Homework checks. 


74 


2.18 


1.63 


136 


2.46 


1.70 


Waiting for responses. 


74 


3.22 


1.53 


136 


3.45 


1.41 


Guiding partial responses. 


72 


3.26 


1.50 


136 


3.23 


1.51 


Tokens and rewards. 


74 


1.45 


1.06 


136 


1.29 


0.93 


Communicating high expectations. 


73 


3.07 


1.46 


136 


3.07 


1.52 


Small groups with individual incentives. 


74 


1.28 


0.88 


136 


1.15 


0.64 


Students encourage one another to 
achieve. 


74 


1.81 


1.37 


136 


1.82 


1.25 


Group contingencies. 


74 


1.18 


0.69 


132 


1.18 


0.71 


1 1. The teacher uses extrinsic behavioral 
incentives such as: 














Praise. 


74 


2.74 


1.51 


136 


2.66 


1.54 


Tokens and rewards for improvement. 


74 


1.41 


1.03 


135 


1.26 


0.88 


Group contingencies. 


74 


1.14 


0.63 


133 


1.17 


0.70 


12. The teacher provides instruction that is 
appropriate for students’ abilities: 














Efforts by the student lead to success. 


73 


3.68 


1.22 


134 


3.66 


1.20 
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Table 12: Observation Data: Descriptive Statistics 
for Use of Time by Class Type 
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Indicators 


ESS Classroom 


Regular Classroom 


N 


Mean 


SD 


N 


Mean 


SD 


13. Allocated time: 














Necessary time is allocated for 
instruction. 


73 


3.90 


1.35 


134 


4.11 


1.16 


14. Engaged rates: 














The teacher uses effective management. 


73 


4.27 


1.03 


134 


3.93 


1.32 


Students attend to lessons. 


73 


4.30 


1.00 


134 


3.97 


1.10 



Indicator subscales. Each set of items comprising an indicator were added together and 
divided by the total number of items to create subscale mean scores ranging from 1.00 to 5.00. 
Independent t tests were computed comparing ESS classrooms and regular classrooms on these 
four subscales, or indicators. Statistically significant differences were found between ESS and 
regular classrooms in quality of instruction and appropriate level of instruction. Quality of 
instruction was higher in regular classrooms than in ESS classrooms (3.52 versus 3.11), with a 
small effect size (practical meaningfulness) of -0.40. Instruction was more often viewed as 
being at the appropriate level in ESS classrooms, though the absolute values of the mean scores 
(2.83 versus 2.15) suggest that appropriate levels of instruction were not often observed in either 
type of classroom, and were observed significantly less often in regular classrooms. There was a 
fairly large effect size associated with this difference (0.73), indicating that there is not only a 
statistically significant difference, but also a meaningful one in a practical sense. There was not 
a significant difference in use of incentives or observed use of time between ESS classrooms and 
regular classrooms. See Table 13 for more details. 



Table 13: Observation Data: Differences in Indicator Subscales by Class Type 



Subscale 


ESS Classroom 


Reg. Classroom 


df 


t 


P 


Dif. 


d 


Mean 


SD 


Mean 


SD 


Quality of 
Instruction 


3.11 


1.09 


3.52 


0.96 


208 


-2.82 


.005 


-0.41 


-0.40 


Appropriate Level 
of Instruction 


2.83 


1.00 


2.15 


0.87 


207 


.18 


.000 


0.68 


0.73 


Incentive 


2.31 


0.73 


2.37 


0.74 


208 


-0.54 


ns 


-0.06 


-0.08 


Use of Time 


4.16 


0.88 


4.01 


1.05 


205 


1.07 


ns 


0.15 


0.15 




65 



52 



Classroom Environment and Resources Checklist 

The Classroom Environment and Resources (CER) checklist assesses the presence or 
absence of indicators of good classroom environments, as well as the visibility and use of a 
variety of resources. Across 24 schools, 213 classrooms were observed and rated using the CER, 
including 193 observations in the fall and winter site visits and 20 ESS classroom observations 
during the summer site visits. Results are presented below. 

As shown in Table 14, most classrooms, regardless of type, were open, risk-free, cheerful 
and inviting environments with adequate lighting and comfortable temperatures and were free 
from distracting noises and/or interruptions. Generally speaking, when compared to ESS 
classrooms, regular classrooms were much more likely to have posted classroom rules and 
assignments in cheerful and inviting atmospheres. Regular classrooms were somewhat more 
likely to use multi-racial and non-sexist materials, display student work, and be free from 
distracting external noises/interruptions. However, ESS classrooms were much more likely to be 
open, risk-free environments, and somewhat more likely to have distinct activity centers and be 
free from internal noises/interruptions. 

Table 14: Observation Data: Presence or Use of Various 
Environmental Indicators by Class Type 



Indicators 


Percent Present or Used 


ESS Classrooms 


Regular Classrooms 


Use of multi-racial materials 


29% 


34% 


Use of non-sexist materials 


38% 


45% 


Posted classroom rules 


54% 


77% 


Posted assignments 


37% 


66% 


Cheerful and inviting classroom 


70% 


81% 


Distinct activity centers 


33% 


31% 


Adequate lighting 


92% 


95% 


Comfortable ventilation/temperature 


93% 


93% 


Student work displayed 


37% 


42% 


No distracting internal noises/interruptions 


78% 


70% 


No distracting external noises/interruptions 


70% 


72% 


Open, risk-free environment 


91% 


77% 


Number of Classrooms Observed 


76 


137 
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As shown in Table 15, in most of the ESS and regular classes, computers, televisions, 
chalkboards, reference materials, textbooks, overhead projectors, a classroom library, and 
instructional aids/props were present. Approximately half or fewer of the classrooms had 
workbooks/activity books, worksheets, joumals/leaming logs, maps/globes, games/puzzles, 
student-used equipment, hands-on materials, audio resources, or video resources. Science or lab 
tables were visible in very few classes of either type. All but one resource listed on the CER was 
more visible in regular classrooms than ESS classrooms. Workbooks/activity books were 
slightly more common in ESS classrooms than in regular classrooms. 

Although it is important for resources to be available to students, more important is the 
extent to which they are used. As shown in Table 15, textbooks, worksheets, and the chalkboard 
were the most common, and maps/globes, games/puzzles, science/lab tables, video resources, 
and classroom libraries were the least common resources to be used in both types of classes. 
Regular classes were more likely than ESS classes to make use of worksheets, joumals/leaming 
logs, maps/globes, instructional aids/props, science/lab tables, the chalkboard, overhead 
projectors, televisions, computers, student manipulatives, and video resources. However, ESS 
classes did make somewhat more use of textbooks, workbooks/activity books, reference 
materials, and games/puzzles. 

Table 15: Observation Data: Visibility and Use of Resources by Class Type 



Resources 


Percent of Classes 
where Resource was 

Visible 


Percent of Classes 
where Resource was 
Used 


ESS 


Regular 


ESS 


Regular 


Textbooks 


84% 


88% 


53% 


42% 


Workbooks/activity books 


43% 


36% 


22% 


17% 


Worksheets 


51% 


60% 


43% 


50% 


Joumals/leaming logs 


17% 


32% 


11% 


19% 


Classroom Library 


61% 


64% 


4% 


4% 


Reference Materials 


63% 


71% 


9% 


3% 


Map and/or globe 


43% 


45% 


0% 


5% 


Games and/or puzzles 


29% 


31% 


3% 


0% 


Instructional aids/props 


55% 


61% 


12% 


17% 


Science/lab table(s) 


1% 


6% 


0% 


2% 


Classroom Chalkboard 


88% 


96% 


33% 


53% 


Student-used equipment 


22% 


24% 


9% 


9% 


Overhead Projector 


70% 


75% 


4% 


24% 


Television 


84% 


90% 


4% 


8% 


Computer 


95% 


96% 


12% 


16% 


Student manipulatives/hands-on materials 


46% 


48% 


11% 


18% 


Audio resources (tapes, CDs, players) 


51% 


53% 


7% 


7% 


Video resources (tapes, discs, players) 


53% 


53% 


3% 


7% 


Number of Classrooms Observed 


76 


137 


76 


137 




67 



54 



ESS School and Program Description Form 

A total of 22 respondents completed and returned the ESS School and Program 
Description Form distributed in the fall 2001/winter 2002 and summer of 2002. When asked to 
identify their roles, nearly half (46%) of the respondents indicated that they were ESS 
coordinators. ESS coordinator/teacher was selected by 41% of the respondents. The remaining 
respondents indicated that they were the ESS coordinator/principal (9%) or the principal/vice 
principal (4%). 

When asked to specify their type of school, nearly half of the respondents (41%) chose 
high school. Nearly a third (32%) selected elementary school, and more than a fourth (27%) 
chose middle school. Further, most of the respondents (96%) indicated that the ESS site was 
school based, while 4% indicated that the ESS site was community based. 

Respondents were asked to describe the community in which the school was located. 
More than half (54%) described the community as rural; the remaining respondents chose the 
descriptors of suburban and urban equally at 23% each. 

Total student enrollment for the schools ranged from a high of 4,944 students to a low of 
260 students. The average number of students per school was 1,064. Total student enrollment in 
the ESS program ranged from a high of 600 to a low of 35. The average number of students 
enrolled in the ESS program was 158. The total number of ESS teachers in each school ranged 
from a high of 28 to a low of 4. The average number of ESS teachers per school was 1 1 . The 
average size of the ESS class in each school ranged from a high of 25 students to a low of 6 
students. Nearly a third (30%) of the respondents indicated that there were 12 students per ESS 
class. The average number of students per ESS class was 12. 

Respondents were then asked to specify the total number of ESS teachers who were also 
regular classroom teachers. As with the question concerning total number of ESS teachers, 
responses to this question ranged from a high of 28 to a low of 4. Of the respondents, 14% each 
listed the number of ESS teachers who were also regular classroom teachers as 4, 5, and 9. The 
average number of ESS teachers who were also regular classroom teachers was 1 1. 

The number of days per week that the program operated at each site ranged from a high 
of six to a low of one. Almost half (46%) of the respondents indicated that the program operated 
two days a week; 32% reported five days a week. The average number of days per week was 
three. The respondents were also asked to indicate the number of hours per day that the program 
operated. The responses ranged from a high of seven to a low of one. More than half of the 
respondents (54%) indicated that the program operated for one hour per day. The average 
number of hours per day was two. 

Respondents were asked to indicate when the ESS program operated by selecting all the 
response options that applied. Most respondents (86%) reported that the ESS program operated 
after school and more than half (54%) indicated that the ESS program operated during the 
summer. Thirty-two percent reported that the program operated before school and nearly a 
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fourth (23%) indicated that the ESS program operated during the intersession. Nine and 4% of 
the respondents, respectively, selected the remaining options of weekends and evenings. 

The respondents were asked to indicate the year in which the ESS program originally 
started. Forty-two percent of the respondents reported that their ESS program started in 1990, 
more than a fourth (26%) indicated that their program began in 1991, and 10% selected 1992 as 
the starting year. Five percent each selected 1993, 1994, 1996 and 1997 as the year in which 
their program began. 

Respondents were asked two open-ended questions. The first asked them to describe the 
major components of their ESS program and the current level of implementation for each 
component. Most respondents, 21 out of 22 (95%), provided an answer to the first part of this 
question, while only 6 of the 21 (28%) provided a response to the second part. Further, multiple 
replies were given by the respondents in answer to the first part of the question. 

The component mentioned most often by respondents was tutoring in general (25%). For 
example, one respondent wrote, “Tutoring is available in core subjects. Teachers station 
themselves in scheduled classrooms so that students may drop in for assistance and get feedback 
on homework, study for tests, get help doing make-up work for time in classes missed, and any 
other general questions.” 

Next most frequently mentioned were activities related to reading, writing, and math 
(18%). A respondent wrote, “The major component is the assistance of homework and 
remediation in the areas of math, reading, and writing.” The summer school and remediation 
programs were each mentioned in 10% of the comments. One respondent noted, “The summer 
program is designed for students to make up credits for the school year. The program is also 
used for remediation of students in all core areas.” 

Other components mentioned were a kindergarten transition program, second chance for 
success, freshman-bridge, intersession, credit recovery, skills building, portfolio development, 
organization skills, make-up work, mentoring, individualized student assistance, collaboration 
with ESS students and teachers, and closing the gap. 

As for the second part of the question, three respondents (50%) replied that the 
components were fully implemented, one respondent replied that the components had limited 
concentration in some grades, one respondent answered that the components . were in the 
advanced stage, and one respondent replied that the component was being piloted. 

The second open-ended question asked the respondents to describe any unique 
characteristics of their community, school, or student population. As with the first question, 21 
out of 22 respondents (95%) provided an answer to this question. Further, most of the 
respondents (76%) provided more than one response to the question. 

At 16% each, comments most often referred to the socioeconomic status of the students 
and the rural locations of the sites. As one respondent wrote, “[There is a] large difference in the 
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socioeconomic backgrounds of the students.” Another respondent noted, “Ninety-eight percent 
of our students are from a rural setting.” 

The high incidence of students eligible for free or reduced-price meals and the diverse 
locales of the students were noted in 13% and 10% of the comments, respectively. One 
respondent replied, “A large majority of our students are on free or reduced lunch (61%).” In 
relation to the diversity of the student population, one respondent mentioned, “[Our] students are 
from rural areas with no running water to affluent suburban areas.” 

One response given to this question was, “The most unique characteristic about our 
school is a learning model that we use in classrooms throughout our building.” Other answers to 
this question included good parental support, ESL students, the presence of single-parent 
families, a low percentage of minority students, small close-knit community, good community 
support, and the presence of transient students. Two of the respondents answered that there was 
nothing unique that they could note; i.e., one said, “I can think of nothing that makes us unique.” 



ICCM 



At the end of each of the 24 site visits, the data collector teams collaboratively completed 
the Innovation Component Configuration Map (ICCM) after reflecting on data collected via the 
surveys, interviews, observations, and documents. This instrument maps the 15 major 
components of the ESS program within the three broad areas of Student Eligibility (four 
■ components), School-Level Program Design (seven components), and District-Wide ESS 
Program Planning (four components). Most of these major components have three possible 
implementation variations (two components have four possible variations). After all pertinent 
data were analyzed, implementation levels were determined for the components. From that 
process, a picture or set of patterns emerged that maps the component configurations. 

The cover page of the ICCM contained quantitative data related to the school 
observations. The average number of ESS teachers per school was 11 (standard deviation of 6); 
the mode was 5, with the number of ESS teachers ranging from 4 to 25. The average number of 
ESS tutors or aides was 3 (standard deviation of 5); the mode was 0 aides, with the number of 
aides ranging from 0 to 24. The number of ESS students varied widely by school; the average 
was 171, with a standard deviation of 184. The mode was 59 ESS students, with a range of 35 to 
667. Finally, the average number of ESS classes per school was 10 (standard deviation of 6); the 
mode was 5, with a range of 3 to 27. 

Table 16 provides descriptive statistical information (mean and standard deviation) for 
each of the 15 major components within the three broad areas for the 24 schools. Thirteen of the 
components had three possible variations (A, B, and C), which were coded as 3, 2, and 1, 
respectively; a score of 3 was the most ideal variation. The table shows that fiscal management 
in District-Wide ESS Program Planning was rated the highest, at 2.62. This was followed by 
scheduling in School-Level Program Design, at 2.54, and student selection in Student Eligibility, 
at 2.48. Standard deviations ranged from 0.45 for collaborative planning processes (District- 
Wide) to double that at 0.90 for scheduling (School-Level). 
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Table 16: ICCM Descriptive Statistics for Major Components Within Three Areas 



Broad Area 




Standard 


Major Component 


Mean 


Deviation 


Student Eligibility 






Referral Guidelines 


2.46 


0.72 


Student Selection 


2.46 


0.51 


Entry and Exit Process 


2.38 


0.77 


Student Assessment for Eligibility 


2.21 


0.66 


School-Level Program Design 






School Transformation Planning 


2.38 


0.77 


Scheduling 


2.54 


0.51 


Staff Selection 


1.88 


0.90 


Staffing Patterns 


2.29 


0.75 


Instructional Practices in ESS Programs 


3.17* 


0.82 


Organizing and Grouping Students 


2.00 


0.59 


Instructional Resources 


1.92 


0.88 


District-Wide ESS Program Planning 






Collaborative Planning Processes 


1.88 


0.45 


Program Evaluation 


2.50* 


0.98 


Fiscal Management 


2.62 


0.58 


Linkages With Other KERA Strands 


2.00 


0.78 



*For these two components, the scores could range from 1 to 4 (most ideal); for all other 
components, the range was 1 to 3 (most ideal). 



Two of the components had four possible variations (A through D, coded as 4 - 1, with a 
score of 4 as the most ideal variation). Instructional practices in ESS programs had the higher of 
the two means, at 3.17 (School-Level); program evaluation (District-Wide) was 2.50. Standard 
deviations for each component were 0.82 and 0.98, respectively. 

Figures 14 through 16 show the means for each major component within each of the three 
broad areas. Overall, it appears that Student Eligibility received the highest ratings and that 
District-Wide ESS Program Planning received the lowest ratings. 
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Figure 14: ICCM Component Means for Student Eligibility 
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Figure 15: ICCM Component Means for School-Level Program Design 
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Figure 16: ICCM Component Means for District- Wide ESS Program Planning 



Findings by Evaluation Topics and Their Related Questions 



Evaluation Topic One: Identification, Referral, and Assignment of Services 

a. Why do students participate in ESS ? 

Nearly all (95%) of the 151 district coordinators who responded to the district 
coordinator survey indicated that most students were referred to the program because they were 
in danger of failing and needed to improve their academic performance. In addition, 90% of the 
district coordinators interviewed ( n = 18) indicated that the program was intended to help 
students succeed, or improve academically. In addition, 73% or more of both ESS ( n = 225) and 
non-ESS ( n = 297) teachers surveyed indicated that their students received ESS services to 
improve academic achievement and because their students were in danger of failing. Additional 
reasons that students received ESS services included because they were in danger of dropping 
out, needed extended learning time, needed to sustain present levels of performance, and needed 
to improve self-esteem. Thus, student performance and success in school were clearly the 
primary reasons that students received ESS services. 

Interviews with parents of students participating in ESS confirmed this finding, given 
that 22% of the 49 parents who were interviewed indicated that their children were participating 
in the program because they needed to improve their grades in a specific subject and 7% needed 
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to improve their grades in general. Sixteen percent of parents cited the students’ need for 
additional help, 9% attributed enrollment to getting caught up on homework, 7% needed help 
staying on task, and 30% indicated a variety of reasons (including falling behind in schoolwork, 
having to make up tests, because the child wanted to participate, and supplemental reasons). 
Finally, 3% each of parents indicated that their children were in ESS because of parental referral, 
parents wanted their children to do well, or because of heavy schedules. Overall, parents 
perceived the reasons for their children’s participation in ESS to be to improve student 
performance and decrease risk of failure. 

In addition, 19% of the ESS students interviewed indicated that they were participating in 
ESS because they needed to do better in school (7%), had failed a class (5%), needed to improve 
a specific grade (4%), or needed to graduate (3%). Nearly one third (31%) of these students 
participated because they volunteered, indicating a desire to improve grades, get extra help, or 
simply wanting to come to the program as specific reasons for their participation. These data 
confirm that the majority of ESS students participated in the program either because they 
recognized a need to improve their school performance, or because they wanted to do so for 
other unstated reasons. 

b. How are students referred for ESS sei-vices and by whom ? 

According to 87% of the 225 ESS teachers surveyed, teacher recommendations were a 
basis for selecting students for participation in ESS. Other means of selecting students for ESS 
included parent requests (56%), student requests (45%), standardized test scores (9%), and other 
(10%). Similarly, 82% of the non-ESS teachers surveyed indicated that selection was based on 
teacher recommendations, 44% indicated that it was based on parent requests, 33% chose student 
requests, 9% chose standardized test scores, and 8% selected other. Responses to the parent 
questionnaire paralleled these data, with 53% indicating teacher recommendation, 32% 
indicating student self-referral, and 29% indicating that parents referred the students. 

ESS teachers’ (« = 98) interview responses to the question about student selection were 
generally consistent with responses on their surveys. Thus, 24% of ESS teachers stated that 
teacher recommendations were used to identify students, 3% stated that school counselors 
selected students, and 2% stated that administrators referred students to the program. In 
addition, 16% of the ESS teachers responded that struggling learners were identified as needing 
ESS, 8% stated that test scores were used to identify students in need of ESS, and 18% and 14% 
of ESS teachers said selection was based on parents referrals or requests and students’ self- 
referrals, respectively. Only 2% of ESS teachers stated that the program was open to all 
students. 

Thirty-one percent of school ESS coordinators who were interviewed ( n = 23) said that 
referrals were made by teachers, 20% said they were made by parents, 10% said they were made 
by students, and 22% stated that eligibility was based on a variety of things including need and 
recommendations from professional school staff. Further, 25% of the 18 district ESS 
coordinators who were interviewed indicated that eligibility for ESS was determined by 
following district guidelines. Presumably these guidelines recommended that students’ academic 
and test performance be used by teachers and other school personnel to identify students for ESS, 
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because 20% of the district ESS coordinators stated that eligibility was based on a combination 
of factors such that the lowest-performing students were given first eligibility, and 18% stated 
that teacher referrals determined eligibility. Five percent of district coordinators surveyed noted 
that the decisions regarding ESS participation occasionally were based on a parent’s request for 
referral. In addition, 8% of the district level ESS coordinators interviewed said that the decisions 
regarding eligibility were made at the school level, and the remaining 15% of their answers 
varied, including following state guidelines and school councils’ referrals. Finally, when asked 
if there was a formal method of referral, 40% of the district level ESS coordinators interviewed 
indicated that teachers initiated referrals to which parents later consented, 10% said that 
community awareness was sought through advertising, and 10% indicated that other various 
methods were used (e.g., computer databases). 

c. Once the referral has taken place, how are individual student goals determined? 

Establishing individual students’ goals for the program appeared to rely heavily on 
parents and teachers, with some student participation. According to 30% of school ESS 
coordinators who were interviewed, the regular or classroom teacher, who was often also the 
referring teacher, determined the goals for students when he/she identified the students’ needs or 
reasons for referral. Another 12% of the coordinators said that the regular and ESS teacher set 
the goals together, and 6% of the coordinators said that the goals were stated on the referral 
forms. Only 24% of the ESS school coordinators said that students were involved in setting the 
goals for their participation, with 12% involving the regular teacher and student working 
together, 6% involving the students and ESS teacher working together, and 6% involving the 
students working alone. Moreover, although parents weren’t identified by the ESS coordinators 
as being part of the goal-setting process, 58% of the parents who were interviewed stated that 
they were indeed part of this process via collaboration with a teacher, as part of planning for 
college, by closely monitoring their children’s progress in school, and by working with their 
child at home. In addition, 10% of the parents who were interviewed said that they would like to 
be involved in the process but were not. 

Regardless of the apparent lack of participation of students in the goal-setting process, 
only 35% of the ESS students interviewed said that they participated in ESS because someone 
else wanted them to do so. Thus, 17% of the students stated that their teachers suggested their 
involvement in ESS, and 18% indicated that one or both parents had encouraged it. Overall, it 
seemed that students’ goals were heavily influenced by their parents and/or teachers but 65% of 
the students apparently accepted these goals as their own and/or understood why they were 
expected to benefit from ESS. 




75 



62 



Evaluation Topic Two: Profiles of Students Receiving Services 

a. What are the grade levels of students receiving ESS? 

ESS students in the sample schools were asked to provide information about their grade 
levels. A total of 1,195 students replied to this query on the student survey. Nearly half (48%) of 
those responding indicated that they were in the high school grades: 5% in 9th, 14% in 10th, 
another 14% in 11th, and 15% in 12th. ESS students in the elementary grades constituted the 
next largest group (28%), with 3% in 1st, 6% in 2nd, 7% in 3rd, 6% in 4th, and another 6% in 
5th. Slightly fewer (25%) were in middle grades: 10% in 6th grade, 8% in 7th, and 7% in 8th. It 
is interesting to note the increases in ESS students in the 6th grade and in the 10th grade; these 
increases perhaps indicate that students transitioning from the elementary and junior high schools 
may require the additional academic support offered via ESS. 

While nearly half of the students who returned surveys were at the high school level, this 
does not imply that more high school youth were participating in ESS sessions, but rather reflects 
the larger size of the high schools; further, half of the summer visits were conducted at the high 
school level. When comparing the percentage of ESS students at a school to the total student 
enrollment, overall building-level percentages were similar for elementary, middle, and high 
schools — from 16% to 19% for all three levels. 

b. What subgroups (gender, race, etc.) are represented by students receiving ESS? 

Gender data were available for students who completed the student survey in the sample 
school sites. Of the 1,163 students who responded to this demographic item, 52% (// = 603) were 
male and 48% (n = 560) were female. 

When disaggregated by school level, it appeared that the percentages of boys and girls 
attending ESS programs in the sample sites remained fairly stable, with boys’ participation 
increasing slightly at the middle school level. Boys constituted 49% of attendees in elementary 
schools, and girls, 51%. Middle school boys accounted for 53% of participants, and girls 
accounted for 47%. Likewise, 53% of ESS attendees were boys at the high school level, and 47% 
were girls. 

Several interesting discrepancies were apparent at the grade levels, however. For 
instance, in the 1st grade, only 40% of students receiving ESS services at sample schools were 
boys. This overrepresentation of girls is anomalous. Dramatically, for example, 69% of ESS 
participants at the 9th grade level were boys, and only 31% were girls. (It should be noted that 
the numbers of students in these two grades are smaller than the numbers in the other grades 
represented.) Boys were also represented more than girls in ESS programs at sample sites in the 
7th (54%), 8th (57%), and 10th (54%) grades. 
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c. What are the achievement levels, as measured by grades and proficiency levels, of the 
students receiving ESS? 

According to 95% of the 151 district coordinators who replied to the survey mailed to all 
Kentucky district coordinators, most students were referred to the program because they were in 
danger of failing. Ninety-five percent also reported that a common reason why students were 
recommended for ESS was to improve their academic performance. 

On the other hand, of the 837 school ESS coordinators who returned their surveys, only 
76% indicated that students were referred to the program because they were in danger of failing. 
This represents a difference of 19 percentage points between the reports of district and school 
coordinators. Nonetheless, school coordinators (92%) tended to report, similarly to district 
coordinators, that students were recommended to the program to improve their academic 
performance. 

Data gathered during site visits suggested that students referred to ESS were at some risk 
for failure, but not to the degree suggested by respondents to the district and school coordinator 
surveys. Of the 18 district coordinators interviewed during AEL site visits, for example, fewer 
than half (45%) reported that students in their districts were recommended for program 
participation because they were not succeeding academically. Similarly, only 43% of the 23 
school coordinators interviewed during AEL site visits thought that students were referred to 
ESS because their performance was deficient (although an additional 6% reported that students 
were referred because they were not achieving to their potential). 

Interestingly, ESS teachers interviewed during AEL site visits reported much more 
frequently that students were referred because they were struggling academically. ESS teachers 
were asked “What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program in your district?” Of the 98 ESS teachers who replied to this 
question, 23% reported that the program was used to assist struggling learners, 13% to improve 
student performance, another 13% to prevent failure or attrition, 7% to improve performance in 
specific academic subjects, and 4% to improve test scores. ESS teachers, thus, provided a 
different perspective on ESS students.’ performance than did coordinators. 

The 225 ESS teachers who completed an ESS teacher survey tended to corroborate this 
view of student performance. Asked to select from a list the most common reasons students 
receive ESS services, 84% indicated that they were to improve their academic achievement, and 
three fourths (75%) noted that students were in danger of failing. In addition, 11% reported that 
ESS students were in danger of dropping out of school. 

Non-ESS teachers likewise thought that ESS students were referred to the program 
because they were performing poorly. Nearly three fourths (73%) of the total 297 non-ESS 
teachers responding to the survey reported that their students received ESS services because they 
were in danger of academic failure. Seventy-three percent also agreed that students were referred 
to ESS to improve their academic achievement. 
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d. What are the characteristics of students receiving ESS that put them at risk of 
dropping out of school — e.g, low achievement in school, poverty, single-parent homes, etc.? 

Data from the School/Program Description Form revealed some characteristics that may 
put students at risk of dropping out of school. For instance, when asked to describe any unique 
characteristics of their community, school, or student population, respondents most frequently 
noted general issues such as socioeconomic status of students (16%) and rurality (16%); at a 
more specific level, respondents noted a high incidence of eligibility for free or reduced-price 
meals (13%) and diverse locales (10%). 



Evaluation Topic Three: Profiles of ESS Programs and Their Implementation Patterns 

a. What are the major components of the ESS program? 

The major components of the Extended School Services program were identified by 
Kentucky educators in the mid-1990s as part of a research project completed by the Kentucky 
Institute for Education Research (KIER). Through a series of meetings, discussions, drafts, and 
reviews, KIER staff guided teams of Kentucky educators in developing of a series of “innovation 
component configuration maps” for all the major thrusts in the KERA law. These maps were 
based on the work of Gene Hall, Shirley Hord, and others in their Concems-Based Adoption 
Models (CBAM). Gene Hall served as a consultant to the KIER component mapping project. 

Specifically, the component map developed for the ESS program was titled the 
Innovation Component Configuration Map for Extended School Services, or ICCM for short. 
The ICCM depicts the major components of the ESS program in the three broad areas of Student 
Eligibility, School-Level Program Design, and District-Wide ESS Programming Planning. 

There were unequal numbers of major program components within the three broad areas 
named above. In the Student Eligibility area, the components included referral guidelines, 
student selection, entry and exit process, and student assessment for eligibility. In the School- 
Level Program Design area, the components included school transformation planning, 
scheduling, staff selection, staffing patterns, instructional practices in ESS programs, organizing 
and grouping students, and instructional resources. Finally, within the District-Wide ESS 
Program Planning area, the major components included collaborative planning processes, 
program evaluation, fiscal management, and linkages with other KERA strands and other 
supporting programs. In sum, then, there were 15 major components in the ESS program, as 
derived from prior KIER research. 

Then, for each of the 15 major components of the ESS program, there were a series of 
possible implementation variations. There were usually three implementation variations for each 
component, but two components (instructional practices in ESS programs and program 
evaluation) had four possible variations. The component implementation variations were 
arranged from left to right on the page for each component and labeled as “Variation A” through 
“Variation C,” except for the two “Variation Ds.” These implementation variations were ordered 
from most ideal (Variation A) to least ideal (Variation D). The most ideal variation (A) was 
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viewed by the ICCM developers as the implementation of that component that was the best that 
an ESS program could achieve. It follows, then, that the most ideal implementation of the ESS 
program would be one that was implementing all 15 major components at the “A” level. Of 
course, in practical terms, it would be very difficult for any ESS program to achieve this high 
standard. To discover where schools in the state were with their implementation of the ESS 
major components, a variety of evaluation study data sources were inspected. 

b. How does the implementation of the ESS program components vary by stakeholder 
groups? 



During site visit interviews, district ESS coordinators were asked to describe their ESS 
services so as to solicit responses, of a general nature, about their implementation of the major 
components of the ESS program. Eighteen district ESS coordinators provided a total of 82 
responses to the question. These 82 responses were grouped into 17 categories, with the number 
of responses in each category ranging from 15 to 2. Comparing the response categories to the 
ESS major components and their implementation variations showed that eight different response 
categories, including 36 (44%) of the total responses, were related to some variation or other of 
the scheduling component in the School-Level Program Design area. For example, summer 
school was named 12 times (15%) by the district ESS coordinators, followed by afternoon 
programs, Saturday school, a.m. and p.m. services, intersession/breaks, before school, night 
classes, and off campus. The single category with the second-largest number (10 or 12%) of 
responses was labeled “monitor budget,” which relates most directly to variations of the fiscal 
management component in the District-Wide ESS Program Planning area. 

Also during site visits, school ESS coordinators were asked the same question about ESS 
services. Their responses were placed into 13 categories, ranging from 20 to 3 percent of the 
replies. It is interesting to note that seven of those categories can be collapsed and related to 
variations of the instructional practices in ESS program components in the School-Level 
Program Design area. Totaling 63% of the school coordinators’ responses, the seven categories 
were subject/content areas, remediation/ skills help, tutorial help, test preparation, 
writing/portfolio, homework, and higher-order skills. Another 6% of the responses on computer 
lab access were variations of implementation in the instructional resources component in the 
same broad area. 

During the fall/winter and summer visits to ESS programs, 98 ESS teachers were 
interviewed and asked several questions about the implementation of key elements in their 
schools. As expected, these questions related to variations of implementation of major 
components in the area of School-Level Program Design. Several questions also related to the 
scheduling component. When asked about time of day for their ESS program, 52% of the 
teachers replied after school, 26% said summer school, 11% said morning before school, 6% 
replied Saturday, 3% said evening hours, and the remaining 3% said the times were flexible. 
Asked the time in the week that they provide ESS services, 35% of the teachers said Monday 
through Friday, 17% said Tuesday and Thursday, 13% said two afternoons per week, 11% said 
Monday through Thursday, and the remaining responses fit into nine different categories. 
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ESS teachers were asked to describe the staffing of their programs, which related to the 
staffing patterns component in the ICCM. Thirty percent of the teachers’ responses fit into a 
category of elective/volunteer. The next highest category was miscellaneous, and contained 19% 
of the responses in a wide variety of statements. Tied for third at 7% were the two categories of 
teacher applies-principal approves and teachers in grade and content. The eight remaining 
categories had from 6% (coordinator selects) to 3% (seniority rules) of the responses. Teachers 
were asked to state the number of students per class, which also yielded information regarding 
the variations of the staffing patterns component. Thirty percent of the responses were in a 
category of 6-10 students to 1 teacher, 17% were in 2-5 to 1, 17% in 11-15 to 1, 15% in 4-30 to 
1, 10% in miscellaneous, and 8% in 16-20 to 1; 5% of the teachers did not know or were non- 
responsive in their reply. 

During the site visits, the ESS teachers were asked a pair of questions related to the 
instructional practices in ESS programs component. When asked about the curriculum of their 
ESS program, the teachers’ responses fit into 10 categories, with responses ranging from 18% to 
3% of the total. Interestingly, the top 3 categories tied with 18% of the responses and included 
everything, same as regular classroom; reading; and math. At 12%, miscellaneous was the next 
category, followed by 9% for core curriculum, 7% for writing/portfolio, 6% for state curriculum 
requirements, 4% for science, and 3% for social studies/history and English/language arts. When 
asked to describe the key elements of the instructional methods of their ESS programs, teachers 
provided a wide variety of responses fitting into 18 categories with 28% to 1% of the total. At 
28%, the category with the most responses was individualized instruction, which was followed 
by the category of small groups at 13%. The remaining 1 1 categories each had less than 10% of 
the total responses and they included, for example (in order), hands-on/manipulatives; games, 
worksheets, etc.; miscellaneous; variety; reading; computers; discussions; whole group; math 
work; tutoring; homework; and others. 

Finally, the ESS teachers were asked to describe their adaptations to student’s needs* 
which yielded variations of implementation for the organizing and grouping students component. 
Fully 35% of the teachers’ responses fit into the category of individualized instruction. Another 
16% were in a category labeled learner needs (but not individualized). Then, the two categories 
of miscellaneous and appropriate level materials tied with 10% each. Next, 9% of the responses 
fit into a category of special education needs. There were six more categories of variations of 
implementations to meet individual need, each with 5% or less of the responses. 

Also, as part of the site visit data collection, 49 parents of ESS students were interviewed. 
One question they were asked was “What are the best parts of ESS for your child?” The parents’ 
responses to this question provided some additional stakeholder information about the variations 
of implementations of the major components in the ICCM. Almost one third (32%) of the 
parents’ responses fit into a category labeled individual attention. The remaining responses were 
placed in seven other categories, ranging from 25% to 6% of the total. At 15%, the second 
largest category dealt with the positive outcomes of the program on the academic standing of 
their children and was labeled improved academically. With 14% of the responses, the category 
of homework help was third largest, followed by the variety of activities in the ESS program at 
10%; and miscellaneous, also at 10%. The last three categories of parents’ responses were 
flexibility, self-confidence, and good teachers/staff, at 7%, 6%, and 6%, respectively. 
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During the 24 site visits to ESS programs, 109 students enrolled in the program were 
interviewed and asked several questions relating to the implementation variations of the major 
components. For example, related to the component of entry and exit process in the Student 
Eligibility area, students were asked how long they had been in ESS and what subjects they 
studied. Student responses to these two questions were often combined rather than separate, but 
four categories of responses were specific to the length of time in ESS. The category with the 
largest number of responses at 21% was called the first semester or first year. Then, 9% 
responded with second year. Next, at just 1% each, the categories of fourth and fifth year were 
named. In the area of School-Level Program Design, students were asked if their ESS teacher 
was the same as their regular teacher. Fifty-six of the responses were in a category of not the 
same teacher, while 37% were in the same teacher category, and 5% said it was mixed (different 
for one subject, same for another). These different implementations relate to the staff selection 
and staffing patterns components in the ICCM. 

ESS students were asked “What do you learn about in ESS?” The responses helped to 
describe the variations of instructional practices and instructional resources in the School-Level 
Program Design area. Student responses to this question were categorized into 15 groups, with a 
few more of those groups naming a subject and the others naming an instructional practice or 
resource. Regarding the former, the subject categories and percentages of total responses were 
English, language arts (14%); reading (13%); math (9%); science (4%); social studies (4%), 
writing (4%); and multiple subjects (3%). Regarding the latter group (practices and resources), 
categories included various activities (16%), individual help (7%), homework (7%), help on tests 
(2%), and improvements (2%). The remaining responses fit in categories of neutral or negative 
reply. 



Finally, students in ESS programs were asked several interview questions related to 
variations of implementation in instructional practices, organizing and grouping students, and 
instructional resources used in ESS. Students were asked to describe what they did in their ESS 
classes that was different or the same as their regular classes. Only 10% of their responses fit 
into the same category and just 4% of their responses were in the mixed (different and same) 
category. The remaining 86% of the responses were spread over 13 difference categories, such 
as different instruction (21%), different miscellaneous (13%), different subject (9%), different 
games (8%), different time (8%), different homework (5%), plus 6 other categories with less 
than 5% each. Students were asked if the ESS teachers did anything special to teach them in 
ESS class that their regular teachers didn’t and, if so, what. Student responses indicated that, 
indeed, ESS teachers did special things in ESS classes. More than three fourths of the responses 
(79%) were put into 10 categories of different, special things, including more individual help 
(34%), miscellaneous (14%), more informal/fun (6%), math practices (5%), different type 
activities (5%), homework help (4%), draws pictures/diagrams (3%), teaches shortcuts/tips (3%), 
does readings (3%), and incentives/rewards (3%). Students also were asked if their ESS teachers 
told them how well they were doing. Student responses were sorted into 10 categories, with six 
of them having 70% of yes/positive responses and the 29% remaining responses in four 
categories labeled mixed answers (11%), no (7%), doesn’t tell me (6%), and off-target response 
(5%). Of the yes/positive responses, the categories were oral feedback (34%), miscellaneous 
(14%), forms/folders/etc. (11%), yes (6%), talks with parents (3%), and reviews work (2%). 





68 



c. What are the patterns of implementation of the ESS components for the more 
effective ESS users and for the less effective ESS users? 

The patterns of implementation of the ESS components were determined through an 
analysis of the ICCM forms completed by the data collection teams at the conclusion of the 24 
site visits. There were 15 major ESS components on the ICCM; each had at least three possible 
variations (A-C) and two had four possible variations (A-D). For analysis purposes, each 
variation was assigned one point and the points per each completed ICCM were totaled. The 
total scores could range from a minimum of 15 to a maximum of 47; the higher the point value, 
the higher the level of ESS implementation toward the most ideal implementation of the most 
components. (Higher implementation scores often are called “high-fidelity implementations.”) 

Actual scores on the ICCM forms ranged from a low of 24 to a high of 45 on the 47-point 
maximum scale. There were 4 scores in the 40s, 15 in the 30s, and 5 in the 20s. To determine 
the patterns of implementation within these scores, the total implementation scores and the actual 
variation scores were examined simultaneously to discern similar groupings, or patterns. Four 
overlapping patterns of implementation emerged from this analysis: Pattern 1, scores of 24 to 31 
(n = 8); Pattern 2, scores of 33 to 38 (/i = 7); Pattern 3, scores of 39 (n = 5); and Pattern 4, scores 
of 40 to 45 ( n = 4). Given the nature of these patterns, a traditional graphic “map” was not 
generated for this analysis. 

The scores for Pattern 3 were very close to the scores for Pattern 4 and, likewise, the 
scores for some of Pattern 2 were close to the scores of Pattern 3. The distinguishing 
characteristic of the groups, then, rests in the details of the patterns of the implementations of the 
15 major ESS components. Put another way, although the total implementation scores were 
close, the patterns of implementation of the various components differed. For example, Pattern 4 
was the most effective — the ESS implementation with the most fidelity to the ICCM. Pattern 4 
implemented all components at variation A or B except for a single school that had all A and B 
variations plus two components with a C variation. In Pattern 2, the second-most effective ESS 
user group, all schools had implementation patterns consisting of all A and B variations and just 
one C variation. The implementation of components for Pattern 3, next-to-lowest effective users, 
consisted of some A and B variations plus either two or three C variations. Last, Pattern 1, the 
least effective ESS users, had implementation patterns either of some A and B variations plus 
four or more C variations or some A, B, C, and D variations. 

Another way to analyze the differences in the patterns of implementation of the ESS 
components was to look for differences within and across the two groups. That is, in addition to 
the count of how many variations of each level each pattern had, an inspection was made of 
which variations were different both within and across Patterns 4 and 1. Within Pattern 4, all 
four ESS programs were judged to be at the B variation for the collaborative planning processes. 
Also within Pattern 4, three of the four programs were at variation B for the scheduling 
component. Within Pattern 1 (the lowest level of implemention), several trends were noted. For 
example, seven of the eight programs in Pattern 1 were judged to be at variations C or D for the 
program evaluation component. In fact, four of those were at variation D. 
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Next, seven of the eight Pattern 1 programs were at variation C for the staff selection 
component, six of the eight were at variation C for the instructional resources component, and 
six of the eight were at variation C for the linkages with KERA strands and other supporting 
programs. Still within Pattern 1, all eight of the programs were evenly split between either 
variation A or B for the scheduling component and, also, six of the eight programs were judged 
at variation B for the organizing and grouping students component. Finally, when inspecting 
which variations differed across Patterns 4 and 1, the most that can be said is that there were no 
outstanding differences evident. No important differences were evident when comparing the 
Pattern 4 ICCMs to the Pattern 1 ICCMs. Of course, the total scores were very different, but the 
patterns that produced those total scores tended to be similar. Stated differently, which 
components were judged to be most ideal (more points) in one pattern seemed to be the same in 
the other pattern and vice versa for components judged to be less ideal (fewer points). 

To investigate further any possible differences across programs for Patterns 1 and 4, their 
School and Program Description Forms (SPDF) were reviewed. Unfortunately, one of the forms 
in Pattern 4 was missing (the school coordinator suffered an injury during the site visit and never 
submitted the completed form). The most interesting trend that emerged from the SPDF was that 
three of the four Pattern 4 schools were middle schools and the other was an elementary school. 
Also, for the three schools with completed forms, coordinators reported that students enrolled in 
the ESS program were just 9%, 10%, and 14% of the total school enrollment. For locale, two 
reported themselves suburban, and the other reported rural. They reported having six, eight, and 
nine ESS teachers — all of them also were regular classroom teachers. All were well-established 
programs, with two begun in 1990 and the other in 1994. The three Pattern 4 schools had after- 
school programs and just one school also offered weekend and summer programs. Two 
programs reported operating three days each week for about one hour each day. The other 
program reported operating five days a week for about four hours per day (this is the school that 
had the summer program). When describing the major components of their ESS programs and 
the current levels of implementation for each component, there were no trends appearing across 
the responses supplied by the three Pattern 4 schools, except that each offered some type of 
tutoring, remediation, or skills-building activities. Otherwise, the responses were rather 
idiosyncratic. 

The eight Pattern 1 (low implementing) school SPDFs were reviewed for possible trends 
across this group. With eight schools in the pattern, the possibilities for confirming trends were 
lessened and this was borne out in their responses to items on the form. There were three high, 
two middle, and three elementary schools in Pattern 1. Four of these schools reported being in 
rural locales, three in urban locales, and one in a suburban locale. The total student enrollment 
varied widely, as did the percentages of ESS students out of the total enrollment. These figures 
ranged from 6% to 36% (with missing data for two schools). Interestingly, although the number 
of ESS teachers ranged from 4 to 25, the number for six of the eight Pattern 1 schools was 9 or 
less. And, similar to Pattern 4, all ESS teachers were regular classroom teachers. Also similar to 
Pattern 4, most of the ESS programs were well established, with just two of the eight starting as 
recently as 1996 or 1997. Seven of the eight schools had after-school programs, while the other 
was a summer program. Four of the eight also offered before-school ESS programs, five offered 
summer programs (other than the one already named), and four offered intersessions. 
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In terms of days of ESS sessions, four schools offered five days, two schools offered four 
days, and one school offered two or three days. The number of hours of ESS programs per day 
varied from one hour up to six hours for the summer programs. In terms of any trends appearing 
in the responses related to descriptions of the major ESS components and the implementation 
levels for each, just one trend was noted. Seven of the eight programs named either reading, 
math, and writing or core subjects; however, this should not be unexpected at all for ESS 
program major components. 

To summarize the differences across the Pattern 4 (high implementers) and Pattern 1 (low 
implementers) SPDF sheets, the most interesting trends discovered were that three of the four 
Pattern 4 programs were in middle schools, and each of them had a rather small ESS program 
enrollment — ranging from 9% to 14% of the total school enrollment. 



Evaluation Topic Four: Services to Students Placed at Risk 

a. Are ESS programs serving students placed most at risk academically? 

One method to determine whether ESS is serving students placed most at risk 
academically is to examine how students are referred to ESS. Permitting multiple “paths” to 
ESS increases the likelihood that all children in need of ESS services will be identified. Teacher 
recommendation was used in all districts as a selection method. In two thirds of the districts, 
parent requests also served as a route into ESS. In 45% of the districts, students could request 
ESS services. In a third of the districts, standardized test scores were used to determine ESS 
eligibility. 

There are myriad reasons why a student might be considered at risk for failure, with the 
most obvious signal being poor performance in a subject. The main reasons students received 
ESS services were because they were in danger of failing (mentioned by 95% of district and 76% 
of school ESS coordinators), and/or needed to improve academic performance (noted by 95% of 
district and 92% of school ESS coordinators). 

More than 88% of all teachers, school coordinators, and district coordinators who were 
surveyed reported increased academic achievement as a main outcome of the ESS program. Of 
the 576 parents who answered the survey item, 58% said that their children had improved 
understanding of the subject material, 36% reported that their children were passing the subject, 
and 22% remarked that their children were passing the grade as a result of participation in ESS. 

b. Are ESS programs meeting the needs of students placed at risk academically? 

Questions were asked in several of the data collection activities to answer this evaluation 
question. The 49 parents of ESS students who were interviewed during site visits were queried 
about their perceptions of ESS effectiveness. When asked how their children were doing in 
school since participating in ESS, more than half (54%) responded that their children’s 
performance in school had improved, with comments such as “He has improved,” and the child 
had “brought grades up.” Seventeen percent of the parents believed that the instruction and 
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assistance their children received in ESS were helpful, saying, “They work better,” and that the 
child was “able to do homework.” Nine percent of the parents mentioned observable changes, 
such as seeing their children be more on task and organized. The outcomes seemed not just 
related to specific content areas, but also included improved study skills or learning how to learn. 

A wider cross section of parents of ESS students was administered surveys. One 
question asked parents how their children’s performance in school had changed since they began 
participating in ESS. Parents were provided five answer choices, ranging from much better to 
much worse. Of the 565 parents who answered the question, 21% said they thought their 
children were performing much better in school since participating in ESS. More than half of the 
parents (57%) said their children were doing better in school. 

When asked during interviews what they thought their children had gained from the ESS 
program, 23% of the 49 parents commented that their children were giving school more attention 
because school was giving the children more attention, i.e., one-on-one instruction and tutoring. 
One parent shared that his or her child had “gained the most from one-on-one contact.” Another 
said, “she blossomed here.” Better grades were a noticeable outcome mentioned by 15% of the 
parents interviewed. Fourteen percent of parents had witnessed increased self-esteem in their 
children. “His self-esteem has gone up tremendously,” remarked one parent. Ten percent of the 
parents believed their children’s participation in the ESS program had resulted in better 
homework skills (“She is better able to do her homework”); an additional 9% of parents had 
noticed their children having better general study skills and organizational skills (“He has 
developed an understanding of time limits”). Seven percent of parents, however, did not know 
. what their children might have gained from participating in the. ESS program. 

The 225 ESS teachers and 223 non-ESS teachers who returned their teacher surveys rated 
the overall effectiveness of the ESS program at their schools. Forty-one percent of ESS teachers 
and 34% of non-ESS teachers rated their programs as excellent. Approximately half of each 
group (51% of ESS and 54% of non-ESS teachers) rated their ESS programs as good. Six 
percent of ESS and 11% of non-ESS teachers reported their ESS programs were fair. Very few 
teachers rated their programs as poor (2% of ESS teachers and 1% of non-ESS teachers). 

Both ESS and non-ESS teachers were asked in the teacher surveys what the most 
important ESS outcomes were for students. The teachers could provide more than one response. 
Almost all (95% of the ESS and 89% of the non-ESS teachers) reported enhanced academic 
achievement. Two thirds (65%) of the ESS teachers observed increased motivation on the part 
of students; in comparison, 38% of the non-ESS teachers noted increased motivation as an 
outcome. Sixty percent of the ESS teachers and 29% of the non-ESS teachers indicated that 
students had better self-esteem because of participation in ESS. 

District and school ESS coordinators were asked the same question about student 
outcomes. Almost all (99%) of the district coordinators stated enhanced academic achievement 
as an outcome of the program. Approximately two thirds (62%) of the district coordinators 
marked increased motivation as an outcome, and 48% reported increased student self-esteem. 
School coordinator responses were similar, with 98% noting enhanced academic achievement, 
70% reporting increased student motivation, and 56% noting an increase in student self-esteem. 
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On the student survey was the statement, “I am a better student this year,” to which 
students either agreed or disagreed. More than three fourths of the 1,201 students (78%) 
responded that they were better students this year. Eighty-five percent of students agreed that the 
ESS program was helping them this year, and 86% stated that they asked for help in ESS when 
they needed it. Finally, when asked if they liked school, 67% responded affirmatively. 

One item on the parent survey asked parents to list the best features of the ESS program. 
The 461 parents who responded to this question provided a total of 522 discrete responses. 
Twenty-one percent of parents said that the best feature was the extra help the program provided 
to students. Helping students to better understand their schoolwork was a best feature noted by 
15% of the parents surveyed. Said one parent, “My child understands math concepts before 
coming home.” The individualized tutoring was mentioned by 11% of the responding parents, 
with parents making comments such as, “The teachers have more time for one-on-one with 
students.” Eleven percent of parents listed as a best feature of ESS the improved academic 
performance of participating students; an additional 10% appreciated that students had time to 
complete homework, review materials, and make up tests. 

Most of the 49 parents (72%) responded positively when asked during interviews if they 
thought their children’s ESS teachers were helping the children to do their best. Parents made 
comments such as, “Teachers are wonderful,” “ESS teachers do help her to do her best,” and 
“[my child] speaks well of her.” A fifth of the parents (21%) did not know or had not heard 
enough about the ESS teacher(s) to comment. 

ESS students were also asked during interviews whether and how their ESS teachers 
might be different from their regular classroom teachers. Thirty-four percent of students 
interviewed stated that the teacher(s) provided more individualized assistance. Nineteen percent 
responded that there was no difference between teachers. Smaller percentages of students 
provided other types of responses, such as the teacher was more informal and fun, helped with 
homework, provided incentives, and explained using diagrams and pictures. 

The students who were interviewed were asked whether the teachers told them how they 
were doing in the ESS classes. Approximately a third replied that yes, they were provided with 
feedback (“The teacher tells us if we are getting better”). In contrast, 13% replied that they were 
not informed of their progress. Other students provided responses that did not fit exactly into 
those categories, but which tended to indicate that they were receiving some sort of feedback 
from the teachers. 

In the student survey was an item asking students what they liked best about the ESS 
program. The top five responses supported what students in interviews had said about their ESS 
teachers differing from their regular classroom teachers. Thirty-nine percent of students 
responded that they liked the tutoring and individualized instruction available in ESS (“I like 
having extra time one-on-one with the teachers so they can explain things better”). Fourteen 
percent appreciated having the extra learning time that ESS provides (“ESS provides the extra 
time that I need for school. It helps me in my schoolwork and it really truly helps me”). Other 




86 



73 



students (12%) said ESS makes learning fun (“It’s fun because you learn different things”). Nine 
percent liked the time to make up work and tests (“I get to make up work that I am failing on”). 

To further understand whether ESS meets the needs of students placed at risk 
academically, parents were asked in surveys whether there were any problems with ESS. Two 
thirds of parents (64%) reported that there were no problems with the program. Nine percent of 
parents cited insufficient communication between ESS and home. As one parent said, “[They] 
need to keep parents more up to date regarding goals and successes.” Approximately 6% would 
like ESS scheduling to be different — either longer, shorter, and/or more regular. Five percent of 
the parents thought that the ESS program at their children’s schools were not rigorous or focused 
enough (“Does not focus directly on what my child’s weaknesses are”). Five percent of parents 
mentioned inadequate resources, including lack of transportation and sufficient teachers and 
insufficient variety of subjects offered. 

Students were asked in the surveys for suggestions on how to improve the ESS program. 
The 1,115 comments from the 1,055 students who provided responses were categorized into 11 
content themes. The five most common response themes are discussed here. Nineteen percent 
of students would like to see schedule changes made to the program. Some students would like 
the program to be longer, others would like it shorter, and some just want it to be at a different 
time. Seventeen percent replied that the program was fine the way that it was and should not be 
changed. Twelve percent of the students suggested the use of more games and other activities 
that either would make learning enjoyable (“You should mix work with a little fun”) or would 
serve as a reward for completing work (“When everyone gets done with their work have a fun 
activity”). 

An additional 12% of students would like to see more or different subjects offered in ESS 
(“Expand its services especially with intersession and allow more classes than just the core”). 
Ten percent of responding students suggested offering more snacks and/or meals as part of ESS. 
(“I think it would be better if they gave us free snacks and drinks because my parents don’t have 
money to give me every week”). Less frequent responses related to the provision of more 
individualized instruction, having instruction instead of study hall, creating a more positive 
learning environment, allowing group work, and providing extra credit. 

c. Are ESS programs identifying and addressing those factors that place students at risk 
for failure? 

There are challenges in working with students placed at risk academically. A quarter of 
the ESS teachers who were surveyed commented that motivating students and getting students to 
attend ESS were hurdles they faced (“Motivating students while focusing on academic 
challenges”). Attendance and student motivation were challenges mentioned also by non-ESS 
teachers (22%), school ESS coordinators (19%), and district coordinators (12%). 

The emphasis at most schools appeared to be individualized instruction, or tutoring. This 
type of instruction was helping students who needed extra time and assistance to master material. 
A non-ESS teacher noted in the teacher survey, “[ESS] helps students who are not read to or 
worked with at home. Seen improvement due to more one-on-one time.” Students were highly 
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appreciative of the extra attention from teachers that ESS provided. Eighty-five percent of those 
surveyed indicated that ESS was helping them that school year, and 39% of ESS students who 
were interviewed commented that the individual tutoring was helping them. The additional time 
to do make-up work or redo work or tests on which the student received a poor grade also were 
mentioned by 31% of the students who responded to the survey as the aspect they liked best 
about the ESS program. 



Evaluation Topic Five: ESS Implementation Patterns and Outcomes 

a: How does the fidelity of ESS implementation correlate with academic index scores? 

As noted under evaluation question three, four distinct patterns of ESS implementation 
emerged from the ICCM instrument. These implementation scores were used to determine 
Pearson correlation values with academic index scores from the 2000-2001 Kentucky Core 
Content Tests (Kentucky Department of Education, 2002) for the following subjects: reading, 
science, mathematics, writing, social studies, arts/humanities, and practical living/vocational science. 
Table 17 presents correlation values of ESS implementation with each of the above subjects. 

As can be seen in Table 17, correlation values between the ICCM total implementation 
scores and each of the school-level variables were very small, indicating a lack of relationship 
between them. None were significant, which was not unexpected given the small sample size of 
24 implementation scores. 

Table 17: ICCM Implementation Score Correlations With School Variables 



School Variables 


Pearson Correlation 
with ICCM Total 
Implementation Score 


KCCT* Reading - Academic Index 


-.044 


KCCT Science - Academic Index 


-.067 


KCCT Mathematics - Academic Index 


-.010 


KCCT Writing - Academic Index 


.039 


KCCT Social Studies — Academic Index 


-.234 


KCCT Arts/Humanities - Academic Index 


-.018 


KCCT PL/VS - Academic Index 


-.015 


Average Years of Teaching Experience 


.021 


Number of Parent Volunteer Hours 


.077 


Spending Amount per Student 


.233 


Attendance Rate 


.147 


Retention Rate 


-.028 


Number of Drug, Weapon, or Assault Incidents 


.029 



*Kentucky Core Content Test 



Note : ICCM scores could range from 15 to 47; the higher the score, the higher the level of ESS 
implementation; actual scores ranged from 24 to 45. 
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b: How does the fidelity of ESS implementation correlate with school-level variables such 
as retention, discipline, attendance, etc. ? 

Again utilizing the ICCM total implementation scores, Pearson correlations were 
generated for a number of school-level variables, including attendance rate, retention rate, 
average years of teaching experience, the number of drug/weapon/assault incidents, the spending 
amount per student, and the number of parent volunteer hours (see Table 17). Again, these data 
were culled from the school report cards provided on the Internet (Kentucky Department of 
Education, 2002). As noted above, correlation values between the ICCM total implementation 
scores and each of the school-level variables were very small, indicating a lack of relationship 
between them. None of the correlations were statistically significant. 

c. How does the fidelity of ESS implementation distinguish between schools with 
minimum and maximum achievement gaps? 

Each of the site visit schools was identified by KDE staff as having either a minimum or 
maximum achievement gap in overall academic index scores between White and minority 
students. To determine whether the ICCM implementation score could differentiate between 
these two classifications, an independent t test was conducted. The mean ICCM implementation 
score for the 13 schools classified as minimum gap was 36.54 (standard deviation of 5.25); the 
score for the 11 maximum gap schools was 32.45 (standard deviation of 5.97); the mean 
difference was 4.08. With a t value of 1.78, and a significance of .088, this difference 
approached but did not reach statistical significance at the .05 level. 

d: What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within the classroom observations? 

For the ESS classroom observations, eight-minute segments were averaged into one set 
of scores per observation, then aggregated and classified by school into the four patterns of ESS 
implementation (lowest to highest levels of implementation) as defined by the ICCM instrument. 
For the classroom snapshot of the observation, students were coded during a one-minute 
observation as being on task, off task, out of the room, or waiting. One-way analyses of variance 
(ANOVAs) were conducted to determine whether statistically significant differences occurred 
among the mean number of students in each of the activities by implementation pattern. No 
significant differences were found between any of the pattern groups. For the number of on-task 
students, means ranged from 8 to 10 across patterns (standard deviations of 3 to 4); for the other 
three activities, means were all below 1 (standard deviations of 0 to 1). 

Then, attention was given to the target student segment of the ESS classroom 
observations. Again, eight-minute segments were averaged into one set of scores per 
observation, then were aggregated and classified by school into the four patterns of ESS 
implementation. The 27 individual activities that a target student could be involved in were 
grouped into the four main categories of teacher-led, management/organization, student-led, and 
off-task. One-way ANOVAs were conducted to determine whether statistically significant 
differences occurred among the mean number of minutes spent in each of the categories by 
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implementation pattern. While no significant differences were found between any of the pattern 
groups, Figure 17 does show fluctuations across patterns. 
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Figure 17: Observation Data: Percent of Time Spent in Main Categories 
by ICCM Implementation Pattern 



Time spent in teacher-led activities ranged from an average of 21 minutes in Pattern 4 
(highest level of ESS implementation) to 30 minutes in Pattern 3 (second highest level of 
implementation), with standard deviations of 3.46 and 16.44, respectively. Much less time was 
devoted to management/organization, as seen by a high of 5 minutes in Pattern 4 (standard 
deviation of 3.30) to a low of 3 minutes for the other three patterns (standard deviations ranging 
from 1.25 to 1.94). Time spent in student-led activities occurred most often in Pattern 4 with a 
mean of 25 minutes (standard deviation of 7.15), and least often for 16 minutes in Pattern 3 
(standard deviation of 8.72). Finally, less time was devoted to off-task activities, with 8 minutes 
in Pattern 1 (lowest level of implementation, standard deviation of 7.34) and 4 minutes in 
Patterns 2 and 4 (standard deviations of 2.33 and 3.93). 

e. What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within selected data measures? 

The following analyses are further investigations of what factors might be underlying the 
four different patterns of ESS implementation. In particular, attention was given to inspecting 
forces that led to successful ESS implementation and barriers that hindered implementation. 
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Available data from the 24 school coordinators from the statewide survey administration were 
utilized for this analysis. Further, data from ESS teachers’ surveys during the 24 site visits were 
aggregated by school and classified into four patterns of ESS implementation. Figures 18 and 19 
show school coordinators’ and ESS teachers’ response percentages for the seven forces that help 
ESS programs to succeed (respondents could select more than one force, as applicable). The 
seven forces included (1) clear support or mandate from district or other political actions, (2) clear 
support from parents or community, (3) additional financial support, (4) excellent staff development 
and follow-up, (5) excellent relationships among staff, (6) outstanding administration, and (7) other. 




Figure 18: Percent of ESS Teachers’ Survey Responses on Forces Leading 
to Successful ESS Implementation by ICCM Implementation Pattern 



All ESS teachers identified five of the seven forces as aiding their ESS implementation: 
district support, community support, financial support, staff relationships, and administration. 
Interestingly, district support was the only force that teachers within schools in Pattern 4 (highest 
level of implementation) selected more often than teachers at other schools; Pattern 4 had 50%, 
Pattern 1 (lowest level of implementation) had 38%. For the other four forces, ESS teachers in 
Pattern 1 showed higher percentages than Pattern 4 (differences of at least 25%), indicating a 
greater perception that these forces were prevalent in their schools. In comparison, school 
coordinators identified the above forces, along with staff development. Data showed two forces 
(district support and staff relationships) with large differences between Patterns 1 and 4 (75% 
difference in district support and 25% in staff relationships). Generally speaking, coordinator 
responses were more indicative of specific forces aiding ESS implementation than the ESS 
teachers. Within the 20 groupings (five forces identified by both teachers and coordinators by four 
patterns), 13 were 60% or more for school coordinators, compared to 7 for teachers. 
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Figure 19: Percent of School Coordinators’ Survey Responses on Forces Leading 
to Successful ESS Implementation by ICCM Implementation Pattern 



Figures 20 and 21 show respondents’ percentages of indicating which of 10 problems or 
obstacles were encountered in their schools as they implemented their ESS programs. These 
obstacles included (1) problems with state or district regulations; (2) opposition or demands from 
key district, school, or other staff; (3) opposition or demands from parents or community; 
(4) problems with teacher unions; (5) inadequate financial support; (6) inadequate preparation of 
teachers or other school staff; (7) problematic relationships among school staff; (8) student 
transportation; (9) opposition or demands from students; and (10) other. 

ESS teachers selected only two problems — inadequate financial support and student transpor- 
tation. A fourth of teachers within Patterns 1 and 4 each selected inadequate financial support; no 
Pattern 2 or 3 teachers selected this problem. All teachers indicated student transportation was a 
problem; however, Patterns 1 and 4 were equal at 25%. In comparison, school coordinators selected 
four problem areas: inadequate financial support, student transportation, student opposition, and 
other. For Pattern 4, 1 00% of the coordinators indicated inadequate financial support, compared to 
13% of the Pattern 1 coordinators. Similarly, more Pattern 4 coordinators indicated some other 
reason was causing problems (33%), compared to 13% of the Pattern 1 coordinators. For student 
transportation, the trend reversed, with 75% of the Pattern 1 coordinators indicating this was a 
problem, compared to 33% of the Pattern 4 coordinators. Similarly, more Pattern 1 coordinators 
indicated student opposition (50%), compared to Pattern 4 coordinators (33%). Generally speaking, 
school coordinators were more indicative of specific problems hindering implementation than the 
teachers. Within the eight groupings (two problems identified by both teachers and coordinators by 
four patterns), two were 60% or more for school coordinators, compared to none for teachers. 
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Figure 20: Percent of ESS Teachers’ Survey Responses on Problems 
Hindering ESS Implementation by ICCM Implementation Pattern 
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Figure 21 : Percent of School Coordinators’ Survey Responses on Problems 
Hindering ESS Implementation by ICCM Implementation Pattern 
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Next, data resulting from the administration of the AEL CSIQ instrument were 
aggregated by school and classified by ESS implementation pattern for the site visit schools to 
determine whether statistically significant differences in scale scores occurred between patterns. 
The six scales included Learning Culture, School/Family/Community Connections, Shared 
Leadership, Shared Goals for Learning, Purposeful Student Assessment, and Effective Teaching. 
One-way ANOVAs were generated for each of the six scales by the four patterns of ESS 
implementation. One statistically significant difference was found in the School/Family/ 
Community Connections scale (F(3,20) = 3.39, p < .05). Tukey’s HSD was used to pinpoint 
which patterns differed significantly and revealed that only Pattern 4 (highest implementation 
level) differed from Pattern 3 (second highest implementation level). Pattern 3 had a mean score 
of 50.57 (standard deviation of 1.97), compared to a mean for Pattern 4 of 43.32 (standard 
deviation of 4.15) for a mean difference of 7.25. There is a large effect size associated with this 
difference (-2.23), indicating that there is not only a statistically significant difference, but also a 
meaningful one in a practical sense. While only one significant difference was found, Figure 22 
does show slight fluctuations among all six of the scales. 
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Figure 22: Mean AEL CSIQ Scale Scores by ICCM Implementation Pattern 



Finally, Pearson correlations were generated for the six mean AEL CSIQ scale scores of 
the site visit schools with the ICCM total implementation score. The correlations follow: 
Learning Culture, .137; School/Family/Community Connections, -.070; Shared Leadership, .046; 
Shared Goals for Learning, .037; Purposeful Student Assessment, -.031; and Effective Teaching, 
.211. These correlations were all very small, indicating a lack of relationship between the scales 
and the implementation score. None of the correlations were statistically significant. 
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CONCLUSIONS 



A number of conclusions can be drawn from the findings presented within this 
comprehensive evaluation of the statewide Kentucky Extended School Services program. These 
are organized by nine topical areas. 



Student Demographics 

• In terms of the proportion of ESS enrollment to student enrollment, ESS participation is 
fairly equal across elementary, middle, and high school building levels; however, 
participation varies widely at the individual school level. 

• The participation of boys and girls in ESS is roughly equivalent, particularly at the 
elementary level. However, fewer females participate in the program at the middle and 
secondary levels. This warrants further investigation to determine whether middle and 
high school girls need fewer ESS services or if they are simply less interested than boys 
in ESS participation. 

• Students attending ESS programs are characterized by coming from poorer areas (rural 
and inner city), which lack resources. These circumstances place students at risk of 
academic failure and dropping out of school. 



Adherence to Intended Goals 

• Generally, students are referred to ESS because they are not performing well 
academically and may be in danger of failing. Other reasons noted were to extend 
students’ learning time, sustain students’ current levels of performance, or improve 
students’ self-esteem. Some students taking advantage of ESS services do so because 
they are in jeopardy of failing at least one class or subject. Thus the achievement of most 
ESS students is depressed when they first begin participating in the program. 

• There is a great deal of consistency among the perceptions of coordinators, teachers, and 
parents as to how students are referred to ESS; the majority believe that students are 
referred most often by classroom teachers. However, students report that they most often 
self-select into the program. It may be that students are taking credit for self-selection by 
agreeing to participate in this voluntary program after a teacher or parent has made the 
suggestion. Either approach seems to allow enough flexibility for the intended 
population to become involved with the program. 

• The students’ regular teachers, ESS school coordinators, and ESS teachers most often 
determine individual student goals, with parents and students themselves being involved 
to a lesser extent. Thus students’ goals appear to be heavily influenced by their teachers, 
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yet the majority of students adopt these goals as their own and appear to understand why 
they are expected to benefit from participation in the ESS program. 

• There is congruence among perceptions of the intended and actual outcomes of the ESS 
program. All stakeholders agree that the ESS program is helping students increase their 
academic achievement, pass courses and grades, and decrease school failure. 

• Given the main reasons for referral, and the outcomes perceived by respondents, it is 
evident that the ESS program is operating within its framework and addressing the main 
goals it is intended to accomplish. 



Classroom Instruction 

• ESS and regular classrooms differ on two major dimensions: quality of instruction and 
appropriate level of instruction. Quality of instruction is better in regular classrooms, but 
instructional level is more often appropriate in ESS classrooms. 

• ESS classrooms tend to engage in student-led activities, often involving independent 
seatwork and pair seatwork. Thus a “typical” ESS classroom appears to be one in which 
students work independently on homework and/or make-up tests, receiving 
individualized instruction as needed. One strength of the ESS classroom arrangement is 
that students are receiving the one-on-one tutoring they need and have the opportunity to 
have concepts not mastered retaught to them. 

• While computers are almost universally available in both ESS and regular classrooms, 
very limited use was made of this resource. However, the environmental checklist did 
not differentiate between one or multiple computers, so in classrooms with only a single 
computer, usage may be restricted to teacher purposes. 



Student Outcomes 

• The ESS program appears to be having an impact on student performance. Nearly all 
teachers and coordinators indicate that participation in ESS has led to increased academic 
achievement. Further, parents report increased understanding of subject material by their 
children, that their children are passing a particular subject, or that their children are now 
doing better in school. 

• Parents and students also report improved study skills and increased motivation to learn 
as a result of participation in ESS. Students appreciate having opportunities to make up 
or retake tests. This flexibility for students who either missed a test or performed poorly 
on a test indicates that value is placed on allowing students the opportunity to show what 
they have learned. 
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• For many students, ESS provides a time to receive individualized instruction, to leam 
study skills, and to have learning reinforced through the use of games, visual aids, 
practice, additional time, and incentives. 



Program Strengths 

• The major strengths of the ESS program focus on processes for its implementation and 
outcomes resulting from that implementation. For instance, process-linked supports 
include targeting students as early as possible, dedicated staff, student transportation, 
collaboration between teachers and coordinators, flexible scheduling, low teacher/ 
student ratio, and individualized instruction. 

• There is a high degree of continuity between coordinators’ and teachers’ beliefs about 
key forces that help the ESS programs to succeed. The most critical components for 
successful implementation are strong district- and building-level support. Other critical 
components for implementation success are collaboration and relationships among staff, 
parent or community support, staff development, and financing. 

• Coordinators’ responses confirm that there are numerous successful programs operating 
in many schools in Kentucky. One particular reason given for success was the use of 
innovative and creative ESS methods. 

• One unique strength of the ESS program is its fluidity and flexibility. Student mobility is 
high throughout the program. As a particular problem arises, ESS allows for an 
immediate intervention that focuses on a specific need that can be addressed before it 
becomes chronic and long term. The program does not rely solely on the results of 
annual standardized test scores, which would slow down the process of identification, 
referral, and enrollment. 



Barriers to Maximum Success 

• A variety of topics are viewed both as weaknesses and as strengths, depending on their 
presence or absence. These include student transportation, funding, staff development, 
parental communication, staffing, and student motivation. This suggests that when these 
factors are in place and sufficient, they provide a strong foundation for successful ESS 
implementation. Conversely, the absence or insufficiency of these factors is detrimental 
to maximizing the potential of an ESS program. These issues are more fully discussed in 
the context of weaknesses so that administrators and policymakers can see the 
explanatory comments related to each. 

• Student transportation is a major problem for some schools. The decision to use ESS 
funds to provide public transportation for students is determined by individual school 
and/or district policies. Because the majority of the ESS services offered during the 
regular school year occur after normal school hours, if bus service is not provided then 
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parents must make transportation arrangements for their children. With the combination 
of parental work schedules, a potential lack of transportation for lower-income families, 
and the distance involved for more rural communities, this factor could seriously deter 
participation of some students who might be most in need of such academic services. 

• Staff development related to ESS now seems to be nonexistent, inadequate, or distributed 
unevenly between teachers and coordinators. This may be more problematic for newer 
staff members who are initially becoming involved with ESS and who are not familiar 
with its related philosophies and guidelines, especially since the ESS summer conference 
was discontinued. Further, there is some lack of agreement among school coordinators, 
teachers, parents, and students as to the exact intent and nature of the ESS program. 

• One discrepancy noted among respondent groups involves communication, especially 
with parents. While ESS teachers believe they meet with parents on an as-needed basis, 
parents note that communication with the teacher about their children’s progress is a 
major problem and that they often are not aware of ESS goals. 

• There seems to be some degree of misunderstanding regarding the emphasis on core 
subjects taught in ESS sessions. District coordinators’ perceptions seem to be most 
closely aligned with the parameters of the ESS policy and regulations. 

• Most of the respondents believe that the current number of teachers involved in ESS is 
inadequate for the number of students. Related to this topic is the reported difficulty 

' associated with recruiting, hiring, and retaining a sufficient number of interested teachers 
with appropriate content knowledge and relevant skills for working individually with 
students in the ESS environment. 

• Student motivation is a relevant issue for encouraging participation in the ESS program. 
Although some students are not motivated enough to participate, those who do participate 
tend to become more interested and to improve their academic performance as a result. 
Moreover, students consider the use of alternative, “fun” instructional strategies in the 
ESS classrooms as more engaging. 

• Finally, there is consent among the coordinators and teachers that additional funding is 
necessary to adequately support full implementation of the ESS program. Addressing 
several of the weaknesses noted above would require an increased level of funding to 
provide consistent student transportation, staff development, expanded services in terms 
of hours and/or subjects, and a reduction of the student/teacher ratio. 



Program Fidelity 

• The ESS programs are performing satisfactorily in terms of implementing the majority of 
the 15 major components of the statewide program. The following four components 
seem to be implemented least satisfactorily: staff selection, instructional resources, 

collaborative planning processes, and program evaluation. 
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• There are four types of implementation of ESS programs in terms of their fidelity in 
operating the 15 major components of the program. That is, there are four levels of 
implementation of the ESS program, ranging from high-fidelity implementers to low- 
fidelity implementers. However, these patterns of implementation are very similar across 
the four groups; the main differences are in the levels of implementation of each 
component, as opposed to the differences across the components. Three of the four high 
implementation schools are middle schools with small ESS programs in terms of the 
number of involved students and teachers. In other words, the high-fidelity 
implementation is more an artifact of program scale and building level rather than 
discrete differences in implementation. 



Patterns of Implementation 

• Although there seem to be no discemable operational differences in the four levels of 
implementation, there are some differences in associated measures when compared by 
implementation pattern. The high implementation group consistently spent less time on 
teacher-led activities and more time on student-led instructional activities than any of the 
remaining three groups. 

• When looking at implementation patterns with other data measures utilized in this 

comprehensive evaluation, one other conclusion can be drawn: All the ESS school 

coordinators in the high implementation group pinpointed inadequate financial support. 



Overall 

• One of the most striking conclusions from this comprehensive evaluation of the statewide 
Kentucky Extended School Services program is the marked consistency and high degree 
of corroboration both within and among respondent perceptions and data collector 
observations. 

• Overall, it is concluded that the ESS program is positively perceived by involved 
stakeholders and has been proven to help address the needs of students who are at risk 
academically. However, several areas have been identified in which improvements could 
be made for a more successful implementation of the statewide program. 
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RECOMMENDATIONS 



Based on the findings and conclusions of the statewide Kentucky Extended School 
Services comprehensive evaluation, a number of specific recommendations are offered for KDE 
staffs review and reflection. 

• Some thought should be given to making scheduling adjustments to the after-school ESS 
programs, such as expanding hours of operation or simply staggering scheduled times 
within a week. The demand for additional time in ESS will need to be weighed against 
the possibility of further exacerbating the conflict between ESS sessions and 
extracurricular activities and/or part-time jobs. 

• ESS staff should encourage/facilitate more involvement of parents and students in setting 
goals for individual students. This would help to improve communication between the 
home and school and to ensure that all involved parties share similar goals for individual 
students’ learning — further increasing the likelihood that these goals will be uniformly 
sought, supported, and achieved. In addition, continued communication with parents 
about their children’s progress should be a routine part of ESS program operation. 

• Professional development opportunities should be provided to ESS coordinators and 
teaching staff in the areas of staff selection, instructional resources, collaborative 
planning processes, individualized instruction, mentoring/tutoring, and program 
evaluation. The specific format for these professional development opportunities could 
vary from workshop sessions at a central site or decentralized sites to online, Internet- 
based courses. Whatever delivery method is selected, professional development in these 
four areas is needed by most ESS program staff in the state. 

• School-level ESS staff should carefully consider the scale of the program as they plan, 
deliver, and evaluate their programs to improve the level of implementation. Rather than 
resorting to downsizing, ESS staff need to assess how thoroughly and effectively they 
have implemented the 15 major program components and develop an action plan for 
improving those areas identified as being low or poor. 

• KDE staff and state board of education members should collaborate to identify possible 
solutions to transportation issues. Solutions might include working closely with 
transportation staff, investigating alternative funding formulas such as using non-ESS 
monies for transportation expenses and/or seeking additional funds specifically for 
transportation. 

• Some thought should be given to exploring ways to overcome the teacher staffing issue. 
For example, KDE staff could identify those districts experiencing ESS teacher 
recruitment problems and work with them to develop solutions. If the problem is teacher 
pay for ESS sessions and state or local regulations that prevent increasing teacher 
salaries, perhaps KDE staff could be instrumental in finding ways to overcome those 
barriers, such as seeking waivers for current rules or regulations. 
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• The possibility of developing an incentive program for . ESS teachers that would generate 
opportunities for recognition of their efforts should be investigated. For example, an ESS 
Teacher of the Year award program might be designed and implemented. The idea is to 
offer a significant award and possibly a financial reward based on state-established 
criteria. The award, which could be regional or statewide, may help draw teachers 
previously uninterested in participating in the ESS program. 

• The summer conference for ESS coordinators and teachers should be re-instituted. This 
conference provides an excellent opportunity for numerous professional development 
sessions for ESS coordinators, teachers, and staff from any district. Also, the 
opportunities to share ESS program information, successes, and solutions to common 
problems would be greater at a large conference. The added value would be that 
professional associations and networking about ESS across the state would likely evolve 
from such a conference; for example, a statewide organization of ESS professionals. 

• It should be clearly communicated to all stakeholders that the ESS program, as 
implemented under current laws and regulations, is not designed to be an enrichment 
program. A clear understanding of the specific nature and purpose of the statewide 
program may help avoid efforts to shift its focus from struggling learners to all students. 

• The current mechanism of categorical funding for the individual ESS programs should be 
maintained. Nearly all district and school respondents agreed this system worked well 
and felt that funds were distributed equitably. 

• KDE and local ESS school staff should investigate ways to recruit at-risk and hard-to- 
reach students. Identified successful methods could be included in the best practices 
resource described below. Schools or districts could apply for grant money to fund 
focused, intensive efforts to increase students' awareness of and interest in the ESS 
program. Other possibilities include modifying current ESS activities to make them more 
fun for students by introducing creative, innovative instructional strategies to better 
capture students' interest, or experimenting with an incentive system to provide more 
extrinsic, short-term rewards to give students a sense of accomplishment during their 
participation in the ESS program (in addition to the intrinsic, long-term goal of increasing 
their academic achievement). 

• KDE staff should formalize and fund the process for obtaining ESS “best practices” and 
develop a resource tool that would be available to all ESS staff. ESS staff in one or more 
districts could be financially compensated for spearheading the initiative and gathering 
submissions from all ESS programs. The final product could be in print or electronic 
format and would be a compendium of innovative and creative ESS programs. It could 
also include a segment on student motivation, as mentioned earlier. We understand that 
such an effort is currently under way, but statewide coordinators indicated limited 
awareness of this undertaking. Therefore, at the very least, KDE staff should increase the 
visibility and potential utility of such a tool for the ESS program statewide. One potential 
resource is the Promising Practices in Afterschool (PPAS) Web site, which provides 
detailed descriptions of promising practices nationwide (see www.afterschool.org). 
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ESS Project: School Selection 

Because ESS is a program designed to provide additional and timely 
instruction to students who need more time to meet achievement goals, one 
measure of the effectiveness of an ESS program within a school is a lack of 
wide variations in performance among subgroups within a school. Ideally, 
no child would be left behind: teachers would regularly assess each student 
for mastery of key skills and content and obtain additional help, including 
ESS, for students struggling to keep pace. 

To identify schools for further analysis as to the effectiveness of ESS an 
intervention program, the KDE data on student performance for 1 999-2000 
was analyzed. (Data for 2000-2001 is not available.) The data file used 
contains performance data on numerous subgroups of students in each 
school. The subgroups relate to gender, ethnicity, socio-economic status and 
participation in various programs, including ESS. On each student’s test 
form, a teacher or administrator at the school were asked to identify the 
student’s participation in various programs in the school. More than half the 
schools in the state identified students as participating in ESS. 

At each school level (elementary, middle and high), this procedure was used 
to select a pool of schools for analysis: 

1) Schools without CATS scores for ESS participants were eliminated; 

2) Schools with relatively small populations of students qualifying for 
the free and reduced lunch program were eliminated; at the 
elementary level, only schools with more than 25% free/reduced 
lunch eligibility were included; the thresholds for middle and high 
schools were 20% and 10% respectively. 

3) The schools were rank ordered from highest to lowest based on the 
overall academic index score for all students. (The academic index 
includes scores for reading, math, social studies, science, writing, arts 
and humanities, and practical living/vocational studies.) 

4) The additional data for these schools includes academic index scores 
for ESS participants, free/reduced lunch participants and African- 
American students, plus these additional indicators: percentage of 
students in the school participating in free/reduced lunch, percentage 
of African-American students, the percent of novice level readers in 
2000, the reduction in the percentage of novice and apprentice 
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readers (novice only in middle schools) from 1994 to 2000, the 
school’s accountability status, the retention rate and, for high schools, 
the dropout rate. 

5) Schools were then placed in two groups: 

a. Schools where students in ESS, free/reduced lunch students and 
African-American students were all scoring within 10 points of 
the school average; 

b. Schools where students in the same subgroups were scoring 
more than 1 0 points below the school average (a few schools 
with ESS scores well above the state average, but free/reduced 
lunch students and minority students scored well below the 
average were also included). 

6) Finally, schools that are generally representative of Kentucky schools 
and students on the basis of geography and demography were 
selected; and no more than one school (except for Jefferson and 
Fayette County elementary schools) was chosen from any district in 
any of the sub-categories. 

Thus, all the schools on the following lists are relatively high performing 
schools based on their overall school scores. The first group of schools is 
also relatively successful with minority and economically disadvantaged 
students. The other group of schools has been relatively successful with 
some students but has not been as successful with minority and 
economically disadvantaged students. 
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Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

flie Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

J 1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


J2 


Evaluator Credibility 


X 








J3 


Information Scope and Selection 


X 








J4 


Values Identification 


X 








J5 


Report Clarity 


X 








J6 


Report Timeliness and Dissemination 


X 








J7 


Evaluation Impact 


X 








-1 


Practical Procedures 


X 








P2 


Political Viability 


X 








F3 


Cost Effectiveness 


X 








?1 


Service Orientation 


X 








P2 


Formal Agreements 


X 








P3 


Rights of Human Subjects 


X 








P4 


Human Interactions 


X 








P5 


Complete and Fair Assessment 


X 








P6 


Disclosure of Findings 


X 








P7 


Conflict of Interest 


X 








P8 


Fiscal Responsibility 


X 








A1 


Program Documentation 


X 








A2 


Context Analysis 


X 








A3 


Described Purposes and Procedures 


X 








A4 


Defensible Information Sources 


X 








A5 


Valid Information 


X 








A6 
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Appendix C: 

Statewide Coordinator Surveys 



Exf© 



Kentucky Extended School Services Program: 

District ESS Coordinator Questionnaire 




Please indicate which responses to the following questions most closely 
match the practices of the ESS program in your school district (fill in 
response circles completely). AH responses will be kept confidential. 



1. What are the most common reasons that students receive ESS? (select all that apply) 

O In danger of failing O To sustain present level of performance 

O In danger of dropping out O To extend learning time 

O To improve academic achievement O Other: 

O To improve self-esteem 

2. How are most of the district's students selected for ESS? (select all that apply) 



O Teacher recommendation O Standardized test scores 

O Parent request O Other: 

O Student request 



3. What subjects are being taught in the ESS program? (select all that apply) 



O Reading 
O Science 
O Math 



O English 
O Social Studies 
O Other: 



4. How is technology used in ESS classrooms? (select all that apply) 



O Drill & practice/academic games O Productivity tools 

O Curriculum O Instructional simulations 

O Communication tools O Classroom management 

O Research tools O Other: 



5. Did you receive staff development related to ESS? 

6. If you received staff development, was it adequate? 

7. Did school-level ESS coordinators in your district 
receive staff development related to ESS? 

8. If they did, was the staff development adequate? 



Yes No 

O O 

O O 

O O 

O O 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © Kt>E A AEL 2001. 



Yes No 

9. Did ESS teachers in your district receive staff O O 

development related to ESS? 

10. If they did, was the staff development adequate? O O 

11. Did non-ESS teachers in your district receive staff O O 

development related to ESS? 

12. If they did, was the staff development adequate? O O 

13. How often do ESS and regular classroom teachers consult on the design of instruction 
and/or goals? 

O Regular classroom teachers O As needed throughout school year 

teach their students in ESS O Only prior to the start of school 

O Regularly throughout school year O Not at all 

14. How often do ESS and regular classroom teachers consult on student performance? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

O N/A (regular teacher is ESS teacher) 

15. How often do ESS teachers and parents consult on student goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

16. How often do ESS teachers and parents consult on student performance? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 



17. How often do ESS teachers and students consult on student goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

18. How often do ESS teachers and students consult on student performance? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 
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19. What are the most important ESS outcomes for the students? (select all that apply) 

O Enhanced academic achievement O Increased motivation 

O Increased self-esteem O Other: 

O Improved attendance 

20. What forces have helped ESS to succeed in your district? (select all that apply) 

O Clear support or mandate from district or other political actions 
O Clear support from parents or community 
O Additional f inancial support 
O Excellent staff development and follow-up 
O Excellent relationships among staff 
O Outstanding administration (principal/coordinator) 

O Other: 

21. What problems or obstacles have been encountered in implementing ESS in your 
district? (select all that apply) 

O Problems with state or district regulations 
O Opposition or demands from key district, school, or other staff 
O Opposition or demands from parents or community 
O Problems with teacher unions 
O Inadequate financial support 

O Inadequate preparation of teachers or other school staff 
O Problematic relationships among school staff 
O Student transportation 
O Opposition or demands from students 
O Other: 

22. Overall, how would you rate the effectiveness of ESS at your school? 

O Excellent O Fair 

O Good O Poor 

23. Which option for disbursing ESS funds would be better for the students and schools? 
O Provide to districts through the SEEK formula 

O Continue to provide as separate categorical funds allotted to districts 
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24. Why do you believe the option you chose for disbursing ESS funds (see Question 23) 
would be better for students and schools? 



25. What are the major strengths of ESS in your district? 



26. What are the biggest challenges faced by ESS in your district? 



27. What recommendations would you make to improve ESS in your district? 



28. What else should we know about ESS? 



Thanks for your cooperation in completing this survey. Your comments are important to us! 



Kentucky Extended School Services Program: 

School Principal/Building Coordinator ESS Questionnaire 




Please select the best description of your role, your school, and your 
community (fill in response circles completely). 



Role: 

O ESS coordinator 
O ESS coordinator and principal 
O ESS coordinator and teacher 
O Principal/assistant principal 
O Classroom teacher 
O Other role 



School : 

O Elementary school 
O Middle/junior high 
O High school 
O Other building level 



Community: 

O Rural 
O Suburban 
O Urban 



Please indicate which responses to the following questions most closely match the 
practices of the ESS program at your school. AH responses will be kept confidential. 



1. What are the most common reasons that students receive ESS? (select all that apply) 

O In danger of failing O To sustain present level of performance 

O In danger of dropping out O To extend learning time 

O To improve academic achievement O Other: 

O To improve self-esteem 

2. How are most of your students selected for ESS? (select all that apply) 



O Teacher recommendation O Standardized test scores 

O Parent request O Other: 

O Student request 

3. What subjects are being taught in the ESS program? (select all that apply) 

O Reading O English 

O Science O Social Studies 

O Math O Other: 

4. How is technology used in ESS classrooms? (select all that apply) 



O brill & practice/academic games 
O Curriculum 
O Communication tools 
O Research tools 



O Productivity tools 
O Instructional simulations 
O Classroom management 
O Other: 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE A AEL 2001. 



Yes 



No 



5 . 

6 . 

7 . 

8 . 
9 . 

10 . 

11 . 



12 . 



13 . 



14 . 



15 . 



Did you receive staff development related to ESS? 

If you did, was the staff development adequate? 

Did ESS teachers at your school receive staff 
development related to ESS? 

If they did, was the staff development adequate? 

Did non-ESS teachers at your school receive staff 
development related to ESS? 

If they did, was the staff development adequate? 



O O 

O O 

O O 

O O 

O O 

O O 



How often do ESS and regular classroom teachers consult on the design of 
instruction and/or goals? 



O Regular classroom teachers 
teach their students in ESS 
O Regularly throughout school year 



O As needed throughout school year 
O Only prior to the start of school 
O Not at all 



How often do ESS and regular classroom teachers consult on student performance? 



O At least once a week 
O At least once a month 



O Only at report card time 
O Not at all 

O N/A (regular teacher is ESS teacher) 



How often do ESS teachers and parents consult on student goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

How often do ESS teachers and parents consult on student performance? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 



How often do ESS teachers and students consult on student goals? 



O Regularly throughout school year 
O As needed throughout school year 



O Only prior to the start of school 
O Not at all 



0 

ERIC 



116 



1 



16. How often do ESS teachers and students consult on student performance? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

17. What are the most important ESS outcomes for the students? (select all that apply) 

O Enhanced academic achievement O Increased motivation 

O Increased self-esteem O Other: 

O Improved attendance 

18. What forces have helped ESS to succeed at your school? (select all that apply) 

O Clear support or mandate from district or other political actions 
O Clear support from parents or community 
O Additional financial support 
O Excellent staff development and follow-up 
O Excellent relationships among staff 
O Outstanding administration (principal/coordinator) 

O Other: 

19. What problems or obstacles have been encountered in implementing ESS at your 
school? (select all that apply) 

O Problems with state or district regulations 
O Opposition or demands from key district, school, or other staff 
O Opposition or demands from parents or community 
O Problems with teacher unions 
O Inadequate financial support 

O Inadequate preparation of teachers or other school staff 
O Problematic relationships among school staff 
O Student transportation 
O Opposition or demands from students 

O Other: 

20. Overall, how would you rate the effectiveness of ESS at your school? 

O Excellent O Fair 

O Good O Poor 

21. Which option for disbursing ESS funds would be better for the students and schools? 
O Provide to districts through the SEEK formula 

O Continue to provide as separate categorical funds allotted to districts 
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22. Why do you believe the option you chose for disbursing ESS funds (see Question 21) 
would be better for students and schools? 



23. What are the major strengths of ESS at your school? 



24. What are the biggest challenges faced by ESS at your school? 



25. What recommendations would you make to improve ESS? 



26. What else should we know about ESS? 



Thanks for your cooperation in completing this survey. Your comments are important to us! 



Appendix D: 

AEL Continuous School Improvement Questionnaire 




Continuous School Improvement Questionnaire 



School Name:. 



YourCoJfset? District Name: 



High Achievement 



State:. 



Date: 



m 

m 



I 



i 

I 









£ 
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Directions: 

This questionnaire concerns your perceptions of your school and community. Therefore, items should 
not be discussed prior to completion. There are no right or wrong answers. Please read each state- 
ment carefully and then rate the extent to which you believe it is true for your school. Using a scale 
of 1 to 6, with a "1" indicating that the behavior is "not present" and a "6" indicating that the behav- 
ior is "present to a high degree," completely fill in the appropriate bubble for each item. 



Like this: 



Not this: © 



Not 




Present to 




Present to a 


present 




some extent 




high degree 


® 


© 


© 

© 


© 


© 



School leadership is proactive in communicating assessment 
results to parents. 



The school works with community groups to improve learning. . 
Assessment data are used to improve student performance. • • • 



School leaders facilitate others in seeking solutions to problems, 
rather than prescribing solutions. 



Groups have been established at this school to participate in 
shared decision making. 



Mistakes are viewed as opportunities for learning rather than 
failures 

Periodically, new goals are adopted to replace old goals. . . . 



The goals of this school are referred to frequently in conversations 
around this school 

Assessment of student learning is directed to improving (rather 
than just monitoring) student performance 



10. School goals focus attention on priorities 

11. The assessment program is based on standards or clear criteria for 

student performance 
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© 



Not 

present 

0) 



Present to 
some extent 

© ® 




Present to a 
high degree 




12. Teachers collaboratively interpret assessment results. 

13. Instructional methods promote student motivation. • • • 

14. School goals focus on results for students 

15. All students are considered capable of learning 

16. Teachers interpret student assessment results for parents 

17. Our school community frequently talks about ways to improve 

student assessment 

18. Teachers use a variety of instructional practices 

19. School goals are stated so that they provide guides for action to 

those who provide instruction 

20. Teachers incorporate opportunities for higher-order thinking. • • • - 

21. The school is proactive in interpreting assessment results for 

members of the community. 

22. The use of assessment data is addressed before assessment 

procedures are established 

23. Classroom practice — what is taught and what is tested is tied to 

schoolwide goals. 

24. The school intentionally selects just a few goals on which to 

focus. 

25. The principal does more listening than telling. . . 

26. Teachers seek feedback from other teachers to improve their 

teaching. . . • • * 

27. Teachers use instructional practices that stimulate curiosity. 

28. Opportunities are provided for teachers to develop leadership 

skills. 



29. Members of the community who are not parents are also involved 



in the school. 

30. Individuals continually look for ways to improve their own 
performance 
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Not 

present 





Present to 
some extent 

© © 




Present to a 
high degree 




31. Teachers are striving to improve their own effectiveness 

32. School leaders provide adequate support to carry out ideas 

33. Teachers provide opportunities for students to reflect on their 

learning. 

34. The principal's leadership style is characterized by "delegating 

power to" rather than "exercising power over." 

35. School staff work actively to stay informed about the community. 

36. Teachers have time to reflect upon their teaching. 

37. In this school, parents are valued as members of a learning 

community. 

38. School leaders use input from others in seeking solutions 

39. Multiple channels of communication keep segments of the school 

community well informed 

40. School staff engage parents as partners in their children's 

education. ■ 

41. The principal listens to all points of view 

42. Teachers in this school are open to innovation. 

43. Decisions regarding use of resources — time, money, space, etc. — are 
made by reference to how well they contribute to schoolwide goals. 

44. The school administration believes in shared leadership. 

45. Teachers help students develop a feeling of being able to control 

their own futures. 

46. Students are challenged to think critically • • 

47. Teachers engage students in problem-based learning. 

48. The principal strives for meaningful community involvement in our 

school. 

49. The purpose of the assessment program is communicated to 

parents. ■ 
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BEST 



Not 




Present to 




Present to a 


present 




some extent 




high degree 
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50. Instructional practices enable students to engage in self-directed 
learning 



51. School goals are reassessed annually. 

52. Students interact frequently during the learning process. • • • • 

53. Teachers are searching for ways to improve their teaching. • • 

54. Members of the community are well informed about the school. 

55. Teachers question established instructional procedures for the 

purpose of improving student learning. 



56. Both teachers and school administrators support instructional 
experimentation 



57. Teachers, administrators, parents, and students work as a team to 
foster learning at this school. 



58. Schoolwide goals for student learning motivate teachers to provide 

good instruction 

59. Administrators include teachers in the decision-making process. • • 

60. Parents feel positive about our school. 
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Appendix E: 



Site Visit Surveys 



Kentucky Extended School Services Program: 

ESS Teacher Questionnaire 




Please select the best description of your role, your school, and your 
community. Fill in response circles completely, like this: • 



Pole: 

O ESS coordinator 
O ESS coordinator and principal 
O ESS coordinator and teacher 
O Principal/assistant principal 
O Classroom teacher 
O Other role 



School: 

O Elementary school 
O Middle/junior high 
O High school 
O Other building level 



Community: 

O Rural 
O Suburban 
O Urban 



Please fill in the appropriate bubble(s) for each item. All responses will be kept confidential. 

1. How many students are in your ESS class? ®©©®©©®©®® 

(If less than 10, the top line should be "0.") ®©©®©©®©®® 

2. How are most of your students selected for ESS? (select all that apply) 

O Teacher recommendation O Standardized test scores 

O Parent request O Other: 

O Student request 

3. What are the most common reasons your students receive ESS? (select all that apply) 

O In danger of failing O To sustain present level of performance 

O In danger of dropping out O To extend learning time 

O To improve academic achievement O Other: 

O To improve self-esteem 

4. What subjects are being taught in the ESS program? (select all that apply) 

O Reading O English 

O Science O Social Studies 

O Math O Other: 

Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE <& AEL 2001. 
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Yes No 



5. Did you receive staff development related to ESS? O O 

6. If you did, was the staff development adequate? O O 

7. How frequently do you consult with regular classroom teachers on the design of 
student instruction and target goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

O N/A (I am the regular classroom teacher) 

8. How frequently do you consult with regular classroom teachers on student 
performance and progress? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

O N/A (I am the regular classroom teacher) 

9. How frequently do you consult with parents on the design of individual student goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

10. How frequently do you consult with parents on student performance and progress? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

11. How frequently do you monitor student performance and progress? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

12. How frequently do you consult with students on the design of their individual goals? 

O Regularly throughout school year O Only prior to the start of school 
O As needed throughout school year O Not at all 

13. How frequently do you consult with students on their performance and progress? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 



O 
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14. What are the most important ESS outcomes for the students? (select all that apply) 

O Enhanced academic achievement O Increased motivation 

O Increased self-esteem O Other: 

O Improved attendance 

15. What forces have helped ESS to succeed at your school? (select all that apply) 

O Clear support or mandate from district or other political actions 
O Clear support from parents or community 
O Additional financial support 
O Excellent staff development and follow-up 
O Excellent relationships among staff 
O Outstanding administration (principal/coordinator) 

O Other: 

16. What problems or obstacles have been encountered in implementing ESS at your 
school? (select all that apply) 

O Problems with state or district regulations 
O Opposition or demands from key district, school, or other staff 
O Opposition or demands from parents or community 
O Problems with teacher unions 
O Inadequate financial support 

O Inadequate preparation of teachers or other school staff 
O Problematic relationships among school staff 
O Student transportation 
O Opposition or demands from students 

O Other: 

17. Overall, how would you rate the effectiveness of ESS at your school? 

O Excellent O Fair 

O Good O Poor 

18. Which option for disbursing ESS funds would be better for the students and schools? 
O Provide to districts through the SEEK formula 

O Continue to provide as separate categorical funds allotted to districts 
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(For scanning purposes, please keep your responses to the following items inside each box. ) 

19. Why do you believe the option you chose for disbursing ESS funds (see Question 18) 
would be better for students and schools? 



20. What is the name of your school? 




21. What are the major strengths of ESS at your school? 



22. What are the biggest challenges faced by ESS at your school? 



23. What recommendations would you make to improve ESS? 



24. What else should we know about ESS? 



Thanks for your cooperation in completing this survey. Your comments are important to us! 




228 




Kentucky Extended School Services Program: 

Non-ESS Teacher Questionnaire 



Please select the best description of your role, your school, and your 
community. Fill in response circles completely, like this: 9 



Role: 

O Principal/assistant principal 
O Classroom teacher 
O Other role 



School : 

O Elementary school 
O Middle/junior high 
O High school 
O Other building level 



Community: 

O Rural 
O Suburban 
O Urban 



Please fill in the appropriate bubble(s) for each item. All responses will be kept confidential. 

1. How many of your students are in ESS? ©o©®©©®©®® 

(If less than 10, the top line should be "O.") ©O©®©©®©®® 

2. How are most of your students selected for ESS? (select all that apply) 

O Teacher recommendation O Standardized test scores 

O Parent request O Other: 

O Student request 

3. What are the most common reasons your students receive ESS? (select all that apply) 

O In danger of failing O To sustain present level of performance 

O In danger of dropping out O To extend learning time 

O To improve academic achievement O Other: 

O To improve self-esteem 

4. In what subjects are your students receiving instruction in the ESS program? 

(select all that apply) 

O Reading O English 

O Science O Social Studies 

O Math O Other: 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE A AEL 2001, 
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Yes No 



5. Did you receive staff development related to ESS? O O 

6. If you did, was the staff development adequate? O O 

7. How frequently do you consult with ESS teachers on the design of student 
instruction and target goals? 

O Regularly throughout school year O Only prior to the start of school 

O As needed throughout school year O Not at all 

8. How frequently do you consult with ESS teachers on student performance and progress? 

O At least once a week O Only at report card time 

O At least once a month O Not at all 

9. What are the most important ESS outcomes for your students? (select all that apply) 

O Enhanced academic achievement O Increased motivation 

O Increased self-esteem O Other: 

O Improved attendance 

10. What forces have helped ESS to succeed at your school? (select all that apply) 

O Clear support or mandate from district or other political actions 
O Clear support from parents or community 
O Additional financial support 
O Excellent staff development and follow-up 
O Excellent relationships among staff 
O Outstanding administration (principal/coordinator) 

O Other: 



O 
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11. What problems or obstacles have been encountered in implementing ESS at your 
school? (select all that apply) 

O Problems with state or district regulations 
O Opposition or demands from key district, school, or other staff 
O Opposition or demands from parents or community 
O Problems with teacher unions 
O Inadequate f inancial support 

O Inadequate preparation of teachers or other school staff 
O Problematic relationships among school staff 
O Student transportation 
O Opposition or demands from students 

O Other: 

12. Overall, how would you rate the effectiveness of ESS at your school? 

O Excellent O Fair 

O Good O Poor 

13. Which option for disbursing ESS funds would be better for the students and schools? 
O Provide to districts through the SEEK formula 

O Continue to provide as separate categorical funds allotted to districts 

(For scanning purposes, please keep your responses to the following items inside each box. ) 

14. Why do you believe the option you chose for disbursing ESS funds (see Question 13) 
would be better for students and schools? 
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15. What is the name of your school? 



16. What are the major strengths of ESS at your school? 




17. What are the biggest challenges faced by ESS at your school? 



18. What recommendations would you make to improve ESS? 




19. What else should we know about ESS? 



Thanks for your cooperation in completing this survey. Your comments are important to us! 
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Ex/ e , 



Kentucky Extended School Services Program: 
ESS Parent Questionnaire 




Please respond to each of the following questions related to the ESS program in which your 
child is enrolled. All responses will be kept confidential. Completely fill in the bubble for the 
appropriate response or write in your answer, as needed. Fill in bubbles like this : • 



What grade is your child in this year? 






0 


1st grade 


0 5th grade 


0 


9th grade 


0 


2nd grade 


0 6th grade 


0 


10th grade 


0 


3rd grade 


0 7th grade 


0 


11th grade 


0 


4th grade 


0 8th grade 


0 


12th grade 



2. Who decided that your child should be in the ESS program this year? 

O I did O The teacher did O My child did 

3. How has your child's performance in school changed since his/her participation in ESS? 

O Much better O Worse 

O Better O Much worse 

O No change 

4. What has your child gained from participating in ESS? (select all that apply) 

O Passing a subject O Improved understanding of the subject 

O Passing the grade O Sets along better at home 

O Will graduate from high school O Sained confidence 

5. How often are you notified of your child's performance in ESS? 

O Every day O Once in a while 

O Every week O Never 

O Every month 

Continue 
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6. How often are you consulted about the goals for your child in the ESS program? 

O Regularly throughout school year O Only prior to the start of school 

O As needed throughout school year O Not at all 

7. Do you understand the ESS program? 



O Fully understand it 
O Understand it somewhat 



O Understand it a little 
O Don't understand it at all 



(For scanning purposes, please keep your responses to the following items inside each box.) 

8. What is the name of your child's school? 



9. What are the best features of the ESS program? 



10. What are any problems with the ESS program? 



11. Do you think that your child should continue in the ESS program next year? Why or why not? 



Thanks for your cooperation in completing this survey. Your comments are important to us! 



0 



134 



SPECIAL INSTRUCTIONS 

for Student Survey 



If the student is in the third grade or lower, or if the student is a 
seriously impaired reader, the data collector will need to work with the 
school coordinator to make special arrangements. For instance, the 
classroom teacher could read aloud each question to the class if third 
grade or lower or for a reading-impaired student at a higher grade 
level. Depending on the student's writing ability, either the student or 
the teacher should record the student's responses. For items 3-12, 
which have a agree/disagree response option, the teacher should 
translate those to yes/no response options when reading items aloud, 
to aid understanding by the student. 



Kentucky Extended School Services Program: 
ESS Student Questionnaire 







Please answer each of the following questions about the ESS program in which you 
participate. All responses will be kept confidential. Completely fill in the bubble for the 



appropriate response or write in your answer, as needed. 


Fill in bubbles like this: • 


1 . 


What grade are you in this year? 










O 1st grade O 5th grade 


0 


9th grade 






O 2nd grade O 6th grade 


0 


10th grade 






O 3rd grade O 7th grade 


0 


11th grade 






O 4th grade O 8th grade 


0 


12th grade 




2. 


I am a: O Boy 


0 


Girl 










Agree 


Disagree 


3. 


I like school. 




0 


0 


4. 


I am a better student this year. 




0 


0 


5. 


I go to school more often this year. 




0 


0 


6. 


I ask for help in school when I need it. 




0 


0 


7. 


I pay attention to my teachers. 




0 


0 


8. 


My parent(s) ask me about school. 




0 


0 


9. 


I attend ESS this year. 




0 


0 


10. 


The ESS program is helping me this year. 




0 


0 


11. 


My ESS teacher lets me know how well I am doing. 




0 


0 


12. 


I ask for help in ESS when I need it. 




0 


0 


13. 


What subjects are you working on in the ESS program? 








O English O Science 










O Mathematics O Social Studies 







O Reading 



O Other 



Continue "t 
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(For scanning purposes, please keep your responses to the following items inside each box.) 



14. What is the name of your school? 




15. What do you like best about ESS? 




16. What would make ESS better? 




Thanks for your cooperation in completing this survey. Your comments are important to us! 
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Appendix F: 

Site Visit Interview Protocols 



Kentucky Extended School Services Program: 
District Administrator Interview 




AEL, an educational research and evaluation corporation in Charleston, West Virginia, has contracted with the Kentucky 
Department of Education to conduct an evaluation of the Kentucky Extended School Services (ESS) program during the 2001-02 
school year. As part of that evaluation, we are interviewing ESS coordinators, teachers, students, and parents to gather their 
perceptions of the ESS program. I'll be recording this interview and taking notes. The taping is simply to accurately record 
your comments to the questions. You will not be identified by your comments; all information summarized will be anonymous. 



Name of district: 

1. How did the school district determine which services it would provide to students? 

2. What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program in your district? 

3. Who designed the program in your district? Did it originate within this district or 
elsewhere? If you got your idea from an outside source, from where? 

4. To what extent were you involved in the design of the program? 

5. Describe the services you offer in the district. 

6. How is eligibility determined? Is this done at the district or school level? 

7. Is there a formal method of referral? If so, how does it operate? 

8. Who determines individual student goals? 

9. How are ESS school-level coordinators selected in your district? 

10. How are ESS teachers selected in your district? 

11. Describe any staff development that teachers (both ESS and non-ESS) have received 
in your district. 

12. Describe any staff development that ESS coordinators (including yourself) have 
received in your district. 

OVER 
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13. Describe communication patterns among the teachers, ESS teachers, and parents. 

a. How often do these parties communicate? 

b. What are the topics of communication? 

c. How does communication influence planning, evaluation, and revisions for individual students? 

14. How does the school district determine success and record the student's progress? 

15. What happens to students who are not doing well in ESS in your district? 

16. What are your assessment procedures? 

17. At what point are students exited from the program? Who makes this decision? 

18. Are there any changes you expect to make during the coming year? If so, what are they? 

19. What is the role of consolidated planning in your ESS program? 

20. Is there anything else you would like to tell me regarding ESS? 
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Kentucky Extended School Services Program: 
School Administrator Interview 




AEL, an educational research and evaluation corporation in Charleston, West Virginia, has contracted with the Kentucky 
Department of Education to conduct an evaluation of the Kentucky Extended School Services (ESS) program during the 2001-02 
school year. As part of that evaluation, we are interviewing ESS coordinators, teachers, students, and parents to gather their 
perceptions of the ESS program. I'll be recording this interview and taking notes. The taping is simply to accurately record 
your comments to the questions. You will not be identified by your comments; all information summarized will be anonymous. 



Name of school: 

1. How did the school determine which services it would provide to students? 

2. What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program? 

3. Who designed the program used at your school? Did it originate within this school or 
elsewhere? If elsewhere, from where? 

4. To what extent were you involved in the design of the program? 

5. Describe the services you offer. 

6. How is eligibility determined? 

7. Is there a formal method of referral? If so, how does it operate? 

8. Who determines individual student goals? 

9. How are ESS teachers selected? 

10. Describe any staff development that you and/or teachers at your school have 
received related to ESS. 



OVER 
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11. Describe communication patterns among the teachers, ESS teachers, and parents. 

a. How often do these parties communicate? 

b. What are the topics of communication? 

c. How does communication inf luence planning, evaluation, and revisions for individual students? 

12. How does the school determine success and record the student's progress? 

13. What happens to students who are not doing well in the program? 

14. What are your assessment procedures? 

15. At what point are students exited from the program? Who makes this decision? 

16. Are there any changes you expect to make during the coming year? If so, what are they? 

17. What is the role of consolidated planning in your ESS program? 

18. How are funds allocated to your school and what steps are taken to ensure that the 
funding formula is equitable? 

19. How does the ESS program fit in as an integral part of KERA? 

20. Is there anything else you would like to tell me regarding ESS? 
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Kentucky Extended School Services Program: 
Teacher Interview 




AEL, an educational research and evaluation corporation in Charleston, West Virginia, has contracted with the Kentucky 
Department of Education to conduct an evaluation of the Kentucky Extended School Services (ESS) program during the 2001-02 
school year. As part of that evaluation, we are interviewing ESS coordinators, teachers, students, and parents to gather their 
perceptions of the ESS program. I'll be recording this interview and taking notes. The taping is simply to accurately record 
your comments to the questions. You will not be identified by your comments; all information summarized will be anonymous. 



Name of school: 

1. Describe the ESS program at your school. 

2. How are students selected for ESS? 

3. Are you a classroom teacher at this school? Are you an ESS teacher at this school? 

4. Describe any staff development you received related to ESS. 

5. How do you coordinate with other staff members regarding ESS? 

6a. If interviewee is both a regular classroom and ESS teacher, ask: 

Do you use the same or different curriculum, methodologies, and materials in your 
regular classroom and your ESS classroom? Why? 

b. If interviewee is only an ESS teacher, ask: 

Do you use the same or different curriculum, methodologies, and materials used in 
your students' regular classroom? Why? 

7. Describe the communications you have with the parents of ESS children. 

8. What are the major strengths of the ESS program at your school? 

9. What are the major weaknesses of the ESS program at your school? 

10. What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program? 



OVER 
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11. Who designed the program used at your school? Did it originate within this school or 
elsewhere? If elsewhere, from where? 

12. To what extent were you involved in the design of the program? 

13. Describe the key elements of the program at your school. 

a. Curriculum 

b. Instructional methods 

c. Staffing 

d. Number of students per class 

e. Adaptations to individual student needs 

f. Time per day and per week that students spend in ESS 

14. How are funds allocated to your school and what steps are taken to ensure that the 
funding formula is equitable? 

15. How does the ESS program fit in as an integral part of KERA? 

16. Is there anything else you would like to tell us regarding ESS? 



Kentucky Extended School Services Program: 
Parent Interview 




AEL, an educational research and evaluation corporation in Charleston, West Virginia, has contracted with the Kentucky 
Department of Education to conduct an evaluation of the Kentucky Extended School Services (ESS) program during the 2001-02 
school year. As part of that evaluation, we are interviewing ESS coordinators, teachers, students, and parents to gather their 
perceptions of the ESS program. I'll be recording this interview and taking notes. The taping is simply to accurately record 
your comments to the questions. You will not be identified by your comments; all information summarized will be anonymous. 



Name of child's school: 

1. What grade is your child in this year? 

2. How important is education to your child? 

3. Why is your child in the ESS program? 

4. How is your child doing in school since his/her participation in ESS? 

5. What do you think your child has gained from the program? 

6. Were you involved in establishing goals for your child? 

a. If you were involved, what was your involvement? 

b. If not, would you like to be involved in the setting of goals for the year? 

7. How are you informed about your child's progress? How often are you informed? 

8. Do you think your child's regular classroom teachers are helping him/her to do his/her best? 

9. Do you think your child's ESS teacher is helping him/her to do his/her best? 

10. What are the best parts of ESS for your child? 

11. What can be improved in the ESS program? 
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Kentucky Extended School Services Program: 
Student Interview 




AEL, an educational research and evaluation corporation in Charleston, West Virginia, has contracted with the Kentucky 
Department of Education to conduct an evaluation of the Kentucky Extended School Services (ESS) program during the 2001-02 
school year. As part of that evaluation, we are interviewing ESS coordinators, teachers, students, and parents to gather their 
perceptions of the ESS program. I'll be recording this interview and taking notes. The taping is simply to accurately record 
your comments to the questions. You will not be identified by your comments; all information summarized will be anonymous. 



Name of school: 

1. Tell me about your ESS program. 

a. What subjects do you study in ESS? 

b. How long have you been in ESS? 

2. Why do you participate in ESS? 

a. Did you choose to participate on your own? 

b. Did your teacher recommend it? 

c. Did your parents recommend it? 

3. What do you learn about in ESS? 

4. Is ESS different or the same as your regular classes? 

a. What do you do in your ESS class that is different than your regular classes? 

b. What do you do in your ESS class that is the same as your regular classes? 

c. Is it easier to learn or study in ESS than in your regular classes? 

5. Tell me about your teacher in ESS. 

a. Is it the same teacher as your regular school day? 

b. Does the teacher do anything special to teach you in your ESS class that your 
regular teacher doesn't do? If so, what? 

c. Does the teacher tell you how you are doing in the ESS class? 

6. Is there anything else you would like to tell me about ESS? 

Adapted from Nesselrodt A Schaffer 2000. © KDE A AEL 2001. 
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Appendix G: 

Special Strategies Observation System 



Kentucky Extended School Services Program: 
Special Strategies Observation System (SSOS) 



© 


© 


© 


© 


© 


© 
















© 


© 


© 


© 


© 


© 










V) 






© 


© 


© 


© 


© 


© 










ft) 

~o 






© 


© 


© 


© 


© 


© 








V) 

ft) 

T5 


3 

4- 

CO 






© 


© 


© 


© 


© 


© 








3 

4- 

CO 


“a 

c 

o 


cn 




© 


© 


© 


© 


© 


© 








*u 


4— 

a 


c 

V 


S— 

ft) 

_c 


© 


© 


© 


© 


© 


© 






S— 


o 

CO 


o 

> 


L 

£ 


4- 

o 


© 


© 


© 


© 


© 


© 


© 


CO 

CO 


O 

3 

cn 

ft) 


O 


O 


O 


o 


© 


© 


© 


© 


© 


© 


© 


LD 


CL 










© 


© 


© 


© 


© 


© 


© 


O 


O 











ft) 

4- 

a 

n 



go 

x> 

i 

o 

5 



c b 
£ "o 



>> a 

ft) 

>- 



c 

ft) 

■O 

3 

4- 

CO 

4- 

a) 

cn 

L_ 

£ 



V) 

V) 

a 



o 

ft) 

Q_ 

fir 



cn 

c 

V) - 

o .> 

‘4= -> 

a — 

§ .§ J> 8 

f U ■§ § 

a 2 S o 

^ a. CL if) 



O O O O 







ERLC 



© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 


© 



t- 

ft) 

> 

£- 

ft) 

V) 

-O 

O 



t- 

ft) 

-Q 

£ 

3 

C 



S- 

O ft) 
o -O 

■5 I 

CO c 



V) 

4- 

C 

ft) 

"O 

3 

4— 

V> 

* 



V) 

V) 

_g 

o 

c 



+1 V) 
3 2 

"o _9 
a o 

tt .£ 



148 



V) 

ft) 

■c ^ E 

.S? o a 

— 4- C 

3 



V) 

4- 

SL 

< 

ft) 

cn 

a 

3 

cn 

c 

a 



tu I I j 

o o o o 



o 

ft) 

In’ 

3 

CO 



a 

> 

t. 

ft> 

to 

-Q 

O 

to 

to 

a 



cn 

ft) 

co 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE A AEL 2001. 



Class Snapshot Ongoing Activities of Target Student 



V) 

w 

o 

Z 



’> 

+■ 

u 

< 

C 

O 

-t- 

c 

ft) 

CL 

V) 

a) 

E 



0) 

■D 

O 

<o 

>- 

‘> 

+■ 

o 

< 



c 

o 

'■»- 

a 

> 

ft) 

V) 

X 

o 



TJ 

c 

<M 



© 

© 

© 

© 

© 

© 

© 

© 



©@o 
© © © 
© © © 
© © © 
© © ® 
© © © 
© © © 
© © © 
© © © 



© 

© 

© 

© 

© 

© 

© 

© 



©@o 
© © © 
© © © 
© © © 
© © © 
© © © 
© © © 
© © © 
© © © 



© 

© 

© 

© 

© 

© 

© 

© 



o©o 
© © © 
© © © 
© © © 
© © ® 
© © © 
© © © 
© © © 
© © © 



© 

© 

© 

© 

© 

© 

© 

© 



© 

© 

© 

© 

© 

© 

© 

© 



© 

© 

© 

© 

© 

© 

© 

© 



o©o 
© © © 
© © © 
© © © 
© © ® 
© © © 
© © © 
© © © 
© © © 



0©O 
© © © 
© © © 
© © © 
© © ® 
© © © 
© © © 
© © © 
© © © 



©@o 
© © © 
© © © 
© © © 
© © ® 
© © © 
© © © 
© © © 
© © © 



o 

ERIC 



§ 

5 

6 
8: 

! 

^0 



c 

o 

V) 

c 

CD 

-O 

3 



o 

L. 

CD 

_D 

E 



© © 
© © 
© © 
© © 
© © 
© © 
© © 
© © 
©© 
© © 



v> 

a 



o 

V) 

c 

CD 

“O 



L_ 

CD 

-Q 

E 



© © 
© © 
© © 
© © 
© © 
© © 
© © 
© © 
©© 
© © 



E 

o 

o 



3 

O 

V) 

-t- 

c 

ft) 

■D 



(_ 

ft) 

X) 

E 



© © 
© © 
© © 
© © 
© © 
© © 
© © 
© © 
©© 
© © 



© © 
© © 
© © 

g) © © 

© © 

© © 
© © 
© © 
©© 
© © 



a 

5 

v> 

+- 

c 

CD 

“O 



V) 

4- 

O 

C_ 

CD 

_Q 

£ 



V) 

.cd 

, <o 

§ 

l 

I 





© © 


© © 




© © 


© © 




© © 


© © 




© © 


© © 


V) 

■H 

C" 


© © 


© © 


CD 

“a 


© © 


© © 


co 


© © 


© © 


4— 

O 

s_ 


© © 


© © 


CD 

-Q 

E 


© © 


© © 


3 

z 


© © 


© © 




© 


© 




© 




VJ 


© 


© 




© 


© 




Stds. 


Stds. 




© © 


© © 




© © 


© © 




© © 


© © 




© © 


© © 


V) 

4- 

r- 


© © 


© © 


CD 

“O 

13 


© © 


© © 


4— 

V) 


© © 


© © 


4— 

O 

L. 


© © 


© © 


CD 

-Q 

E 


0 0 


0 0 


3 

z 


© © 


© © 




© 


© 






® 


SC 

V) 


© 


© 


a 

\- 


0 


© 




Tchr. 


Aide 



14 9 BE §T COPY AVAILABLE 



Adapted from Nesselrodt <& Schaffer 2000. Machine Scannable Version © KDE <& AEL 2001. 



iuiri niiriuie ui uuoci vuiium anuuiu ucym ui* 

Class Snapshot Ongoing Activities of Target Student 



V) 

CD 
















© © 


© © 


4- 

o 

z 
















© © 


© © 


















© © 


© © 


















© © 


© © 
















v> 

4- 


© © 


© © 


>- 

4_ 


© 


© 


© 


© 


© 


© 


CD 

"O 

3 


© © 


© © 


’> 

4- 


© 


© 


© 


© 


© 


© 


4— 

tO 

s- 


© © 


© © 


O 

< 


© 


@ 


© 


© 


© 


© 


o 

t- 


© © 


© © 


c 

o 

4- 


© 


© 


© 


© 


© 


© 


CD 

_Q 

£ 


© © 


© © 


c 

CD 

CL 


© 


© 


© 


© 


© 


© 


3 

z 


© © 


© © 


tO 

CD 


© 


© 


© 


© 


© 


© 








£ 

i— 


© 


© 


© 


© 


© 


© 




© 


© 




0 


0 


0 


© 


© 


© 




© 


© 


















© 


© 
















V) 

£ 


© 


© 




0©O 


0©O 


0©O 


0©O 


0©O 


©@o 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




V) 

"O 

4— 


v> 

"O 

4— 


CD 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




to 


to 


O 

CD 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








>- 

4- 

1E 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 


4— 

U 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








< 


©0© 


©0© 


©0© 


©0© 


© 0 © 


©0© 




W O’ 






© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


© © © 


© © © 


© © © 


© © © 


© 0 © 




© © 


© © 
















V) 

4- 

C 


© © 


© © 



C 

o 

4- 

a 

> 

(D 

V) 

_Q 

O 

s- 

o 

CD 



§ 

I 

I 

I 





© © 




© © 




© © 




© © 




to 

4— 




© © 




© © 




© © 




© © 




O 

s. 

CD 




© © 




© © 


£ 

o 


© © 




© © 




X> 

£ 


V) 


© © 


V) 

a 


© © 


o 

L. 

4- 


© © 


CT> 

c 


© © 




3 

z 


O 

4- 

C 


© © 


4— 

M- 

H- 


© © 


O 

4- 

3 


© © 


4- 

*a 

> 


© © 






o 

V) 

4— 


© © 


O 

V) 

4— 


© © 


O 

V) 

4- 


© © 


V) 

4- 


© © 


.CD 




c 

CD 

~o 


© © 


C 

CD 

"O 


© © 


C 

CD 

“O 


© © 


C 

CD 

“O 


© © 






3 

4— 

V) 


© © 


3 

4— 

V) 


© © 


+“ 

V) 

u 


© © 


V) 


© © 




V) 

a 

i 


H— 

O 

L. 


©© 


o 

L. 


©© 


't 1 

o 

c_ 


©© 


1 “ 
o 
c_ 


©0 


§ 


i 


CD 

-Q 

£ 


© © 


CD 

_Q 

E 


© © 


CD 

-Q 

£ 


© © 


_Q 

£ 


© © 


1 

s> 




3 

z 




3 

z 




3 

z 




3 

z 









- 'o 

ERIC 



© © 
© © 
0 0 
© © 

© 

© 

© 

0 



o 

I- 



© © 
© © 
© © 
© 0 
© © 

© 

© 

© 

© 



CD 

“O 



150 



BEST COPY AVAILABLE 



Adapted from Nesselrodt & Schaffer 2000. Machine Scannable Version © KDE & AEL 2001. 



iuin miriuic ui uudci vunuri oiiuuiu uc-y 

Class Snapshot Ongoing Activities of Target Student 



v> 

&) 
















© © 


© © 


4- 

O 

Z 
















© © 


© © 


















© © 


© © 


















© © 


© © 
















v> 

4- 

r“ 


© © 


© © 


>- 

4— 


0 


0 


0 


0 


0 


0 


CD 

“O 

3 


© © 


© © 


’> 

4“ 


© 


© 


© 


© 


© 


© 


4— 

to 

t+— 


© © 


© © 


O 

< 


0 


0 


0 


© 


0 


0 


O 

L. 


© © 


© © 


c 

o 

4- 


© 


© 


© 


© 


0 


0 


_o 

E 


© © 


© © 


c 

0) 

CL 


© 


© 


© 


© 


© 


© 


3 

z 


© © 


© © 


to 

CD 


© 


© 


© 


© 


0 


0 








£ 

i- 


© 


© 


© 


© 


© 


© 




© 


© 




© 


© 


© 


© 


© 


© 




© 


® 


















© 


© 
















V> 

£ 


© 


© 




©@o 


©@o 


©@o 


©@o 


©@o 


©© G 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




V) 

“O 

4— 


VI 

“O 

4— 


CD 

-n 


0 0 0 


© © © 


© © © 


© © © 


© © © 


© © © 




to 


to 


U 

o 

<0 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








> 


© © ® 


© © © 


© © © 


© © © 


© © ® 


© © ® 




© © 


© © 


4— 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








O 

< 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


0 0 0 


@00 


© © © 


© © © 


© © © 




© © 


© © 
















V> 

4- 

C 


© © 


© © 



C 

o 

'4- 

o 

> 

£_ 

CD 

V) 

_Q 

O 

H— 

O 

CD 

4— 

3 

C 



00 



- ‘O 

ERIC 



4^ 

§ 

§ 

4^ 

! 

<0 



V) 

O 

4— 

C 

o 



c 

<d 

"O 

3 

4— 

V) 

M- 

O 

£_ 

CD 

_Q 

£ 

3 

z 



© © 




© © 




© © 




© © 


© © 




© © 




© © 




© © 


© © 




© © 


£ 

o 


© © 




© © 


© © 


V) 

O 


© © 


VJ 

£_ 

*4— 


© © 


CT> 

t C 


© © 


© © 


4— 

H- 

M— 


© © 


O 

4- 

3 


© © 


4— 

'a 

> 


© © 


© © 


O 

VI 

4- 


© © 


O 

VI 

4- 


© © 


V) 

4- 


© © 


© © 


CD 

“D 


© © 


CD 

"O 


© © 


C 

CD 

“D 


© © 


© © 


3 

4— 

V) 


© © 


4— 

V) 


© © 


4— 
V) 
1 1 


© © 


©© 


H— 

O 

L. 


©© 


H— 

O 

£_ 


©© 


o 

£. 


©© 


© © 


_Q 

£ 


© © 


CD 

_Q 

£ 


© © 


CD 

_Q 

£ 


©© 




3 

z 




3 

z 




3 

z 





to 

H— 

o 

£_ 

CD 

-Q 

e 

3 

z 



v> 

4^ 

vj 

§ 

i 

i 



V> 

|2 



© © 
© © 
© © 
© © 
© © 

© 

© 

© 



o 

H 



© © 
© © 
© © 
© © 

© 

© 

© 



"O 



151 



BEST COPY AVAILABLE 



Adapted from Nesselrodt & Schaffer 2000. Machine Scannable Version © KDE & AEL 2001. 



Class Snapshot Ongoing Activities of Target Student 



V) 

&) 
















© © 


© © 


4- 

O 

z 
















© © 


© © 


















© © 


© © 


















© © 


© © 
















V) 

4- 

r- 


© © 


© © 




© 


© 


© 


© 


© 


© 


CD 

“O 

3 


© © 


© © 


4- 


© 


© 


© 


© 


© 


© 


4— 

H- 


© © 


© © 


< 


© 


© 


© 


© 


© 


© 


O 

£- 


© © 


© © 


C 

o 

4- 


© 


© 


© 


© 


© 


© 


CD 

JD 

£ 


© © 


© © 


c 

<D 

Q- 


0 


© 


© 


© 


© 


© 


3 

z 


© © 


© © 


</) 

G) 


© 


© 


© 


© 


© 


© 








£ 

f- 


© 


© 


© 


© 


© 


© 




© 


© 




© 


© 


© 


© 


© 


© 




© 


© 


















© 


© 
















V) 

a 

h- 


0 


© 




©@o 


o©o 


o©o 


©@o 


o©o 


©@o 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




V) 

“O 

4— 


V) 

"O 

4— 


CD 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




to 


to 


~o 

o 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








>* 

4- 

*> 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 


4- 

O 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 






/O 


< 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 






© © © 


0 © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 
















V) 

4- 

C 

- CD 


© © 


© © 









© © 




© © 




© © 




© © 




4- 

V) 

M— 


© 


© 


© 


© 


c 

o 






© © 




© © 




© © 




© © 




O 

L. 


© 


© 


© 


© 


o 

> 






© © 




© © 


£ 

o 


© © 




© © 




£ 


© 


© 


© 


© 


£_ 

CD 

V) 




22 

V) 


© © 


2*: 

53 


© © 


o 

s_ 

4— 

fS 


© © 


*d> 

c 


© © 




3 

z 


© 


© 


© 


© 


-Q 

O 

H- 

o 




O 

4— 

C 

o 


© © 


4“ 

M- 

M— 

O 


© © 


U 

4- 

3 

O 


© © 


4- 

a 

5 


© © 


V) 




© 




© 




CD 

4- 

3 


■fC 

§ 


V) 

4- 

c 


© © 


V) 

4- 

C 


© © 


V) 

4- 

C 


© © 


V) 

4- 

C 


© © 






© 




© 




C 

c 


£ 

& 


CD 

~XD 


© © 


CD 

-o 


© © 


CD 

-o 


© © 


CD 

■o 


© © 






(?) 




(?) 




-C 

+- 

\0 


ti i 


3 

4- 

V) 

M— 


© © 


3 

4- 

V) 

*4— 


© © 


4- 

V) 

H- 


© © 


4- 

V) 

H- 


© © 




Task 


© 




0 




CNJ 


4^ 

c- 


O 

s_ 


©© 


O 

£_ 


©© 


O 

5- 


©© 


O 

l— 


©© 


§ 

Vi 












1 


CD 

JD 

£ 


© © 


CD 

-O 

£ 


© © 


CD 

X> 

£ 


© © 


CD 

X> 

£ 


© © 


I 




£_* 
-C 
1 \ 




CD 

“O 






<0 


3 

z 




3 

z 




— J 

z 




z 








h- 




< 





■- O ■ 

ERIC 



152 BEST COPY AVAILABLE 



Adapted from Nesselrodt & Schaffer 2000. Machine Scannable Version © KDE & AEL 2001. 



Class Snapshot Ongoing Activities of Target Student 



V) 

(D 


















© © 


© © 


4- 

o 

z 


















© © 


© © 




















© © 


© © 




















© © 


© © 


















V) 

4- 

C 


© © 


© © 


>. 

4— 


© 


© 


© © 


© 


© 




CD 

"O 

3 


© © 


© © 


> 

+* 


© 


© 


© © 


© 


© 




4— 

co 

n 


© © 


© © 


o 

< 


© 


© 


© © 


© 


@ 




o 

£_ 


© © 


© © 


c 

o 

4- 


© 


© 


© © 


© 


@ 




CD 

-D 

£ 


© © 


© © 


C 

CL 


© 


© 


© © 


© 


© 




3 

Z 


© © 


© © 


CO 

ft) 


© 


© 


© © 


© 


© 










£ 

1- 


© 


© 


© © 


© 


© 






© 


© 




© 


© 


© © 


© 


© 






© 


© 




















© 


© 


















V) 

£ 


© 


© 




©@o 


©© O 


©©o ©©O ©©o 


©© O 












© © © 


© © © 


©©© ©@© ©@© 


© © © 






to 

"O 

4— 


to 

"O 

4— 


CD 

n 


© © © 


© © © 


©©© ©©© ©©© 


© © © 






</) 


CO 


o 

<o 


© © © 


© © © 


©@© ©@© ©©© 


© © © 










4- 


© © ® 


© © ® 


©@® ©©® ©@® 


© © ® 










*> 


















© © 


© © 


4— 

f \ 


© ® © 


© © © 


©®© ©@© ©@© 


© © © 










< 


©0© 


©0© 


©0© ©0© ©0© 


© © © 






© © 


© © 




© © © 


© © © 


©©© ©©© ©@© 


© © © 






© © 


© © 




© © © 


© © © 


©©© ©©© ©©© 


© © © 






© © 


© © 


















V) 

4- 

C 


© © 


© © 


















CD 

~o 

3 


© © 


© © 






© © 


© © 


© © 




© © 




4— 

iO 

If 


© © 


© © 


c 

o 




© © 


© © 


© © 




© © 




o 

s_ 


© © 


© © 


4- 

a 

> 




© © 


00 f 


© © 




© © 




JD 

£ 


© © 


© © 


£_ 

CD 

V) 




© © 


"|j © © ^ 


© © 


C 


© © 




3 

z 


© © 


© © 


o 

M- 

O 


o 

4- 

C 

•• o 


© © 


it © © 3 

O O 


© © 


4- 

’a 

5 


© © 


V) 




© 


© 


CD 

4- 

3 


r* v> 

i = 


© © 


t © © 42 

c c 


© © 


V) 

4- 

C 


© © 


.3 






© 


.E 


5 CD 


© © 


| ©© | 


© © 


CD 


© © 


4^ 

r * 






(?) 


c 

-C 


s' 5 


© © 


to © © v> 


© © 


4— 

V) 


© © 


VJ 


V) 

C3 






£ 


1$ "+r 




M— M— 




M- 








© 


© 


CO 


o 

s h 


©© 


z 0 © ° 


©© 


O 

S_ 


©© 


§ 










1 1 


© © 


CD CD 

f ©© -§ 


© © 


CD 

-O 

£ 


© © 


1 




L_* 

_c 


a> 

-o 


‘ A 


Z'* -J 

'o z 




z z 




z 








i— 


< 



ERIC 



153 



BEST COPY AVAILABLE 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE A AEL 2001. 



Class Snapshot Ongoing Activities of Target Student 



V) 
















© © 


© © 


4— 

O 

z 
















© © 


© © 


















© © 


© © 


















© © 


© © 
















V) 

4- 

r~ 


© © 


© © 


>. 

-1- 


© 


© 


© 


© 


© 


© 


CD 

~o 

3 


© © 


© © 


*> 

4— 


© 


© 


© 


© 


© 


© 


4— 

tO 


© © 


© © 


O 

< 


© 


© 


© 


© 


© 


© 


O 

s_ 


© © 


© © 


E 

O 

-1- 


© 


© 


© 


© 


© 


© 


CD 

X 

£ 


© © 


© © 


E 

CD 

CL 


© 


© 


© 


© 


© 


© 


3 

z 


© © 


© © 


If) 

CD 


© 


© 


© 


© 


© 


© j 








£ 

j- 


© 


© 


© 


© 


© 


© 




© 


© 




© 


© 


© 


© 


© 


© 




© 




















© 


® 
















VI 

H 


© 


© 




0©O 


0©O 


0©O 


0©O 


0©O 


0©O 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




VI 

-o 

4— 


V) 

~o 

4— 


CD 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




CO 


CO 


u 

O 

CD 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








>. 

4- 

;> 


© © © 


© © © 


© © ® 


© © ® 


© © ® 


© © © 




© © 


© © 


4— 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








O 

< 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 
















V) 

4- 

E 


© © 


© © 



E 

o 

4- 

a 

> 

CD 

V) 

X 

o 



o 



3 

E 



"D 

C 

CM 

M" 



§ 

I 

I 

<0 



V) 

a 

4- 

E 

o 

V) 

4- 

E 

CD 

“O 

3 

4- 

V) 

4- 

O 

c_ 

CD 

_Q 

e 



o 

ERIC 



© © 
© © 




© © 
© © 




© © 
© © 




© © 
© © 


4- 

<o 

4- 

O 

L. 

CD 


© © 
© © 


© © 




© © 


£ 

o 


© © 




© © 


X 

£ 


© © 


© © 


V) 

a 

4- 


© © 


o 

L- 

4- 

O 


© © 


CXI 

E 


© © 


3 

z 


© © 


© © 


4- 

4- 

O 


© © 


4- 

3 

O 


© © 


4— 

a 

5 


© © 


V) 


© 


© © 


V) 

4- 

E 


© © 


V) 

4- 

E 


© © 


V) 

4- 

E 


© © 


.3 

4^ 


® 


© © 


CD 

13 

3 


© © 


CD 

"D 

3 


© © 


CD 

"O 

3 


© © 




® 


© © 


4- 

V) 

4— 


© © 


VI 

4— 


© © 


4- 

VI 

4- 


© © 


^ 8 


© 


©© 


O 

t- 

CD 


©0 


O 

t_ 

CD 


©© 


O 

t- 

CD 


©© 


§ 

$ 


© © 


X 

£ 

3 

z 


© © 


X 

£ 

3 

z 


© © 


XI 

£ 

3 

z 


© © 


i 


Tchr. 



"D 

3 



© © 
© © 
© © 
© © 

© 

© 

© 



"O 



154 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE 4 AEL 2001. 



Class Snapshot Ongoing Activities of Target Student 



v> 

0) 
















© © 


© © 


4- 

o 

z 
















© © 


© © 


















© © 


© © 


















© © 


© © 
















v> 

4- 

c~ 


© © 


© © 


>- 

+- 


© 


© 


© 


© 


© 


© 


d) 

"O 

3 


© © 


© © 


’> 

+“ 


© 


© 


© 


© 


© 


© 


£ 

M— 


© © 


© © 


u 

< 


© 


© 


© 


© 


© 


© 


O 

S- 


© © 


© © 


c 

o 

+- 


@ 


© 


© 


@ 


@ 


@ 


d) 

-Q 

E 


© © 


© © 


c 

d) 

CU 


© 


© 


© 


© 


© 


© 


3 

Z 


© © 


© © 


If) 

d) 


© 


© 


© 


© 


© 


© 








£ 

i- 


@ 


© 


© 


© 


© 


© 




© 


© 




© 


© 


© 


© 


© 


© 




® 


® 


















© 


© 
















V) 

a 

1— 


© 


© 




0©O 


0©O 


0©O 


©@o 


0©O 


©@o 










© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




+■ 


V) 

TD 

4“ 


d> 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




If) 


V) 


u 

o 

o 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 








4— 

’> 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 


+■ 

u 


© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




/CT\ 




< 


© © © 


© 0 © 


© 0 © 


© © © 


© 0 © 


© 0 © 






Ky 




© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 




© © © 


© © © 


© © © 


© © © 


© © © 


© © © 




© © 


© © 
















V) 

4— 

c 

_ At 


© © 


© © 



C 

o 

4 - 

a 

> 

s_ 

d) 

v> 

X) 

o 



o 

Cl) 

4 - 

3 

E 

£ 

JZ 

8 

ir> 



- o 
ERIC 



4^ 

§ 

S 

4^ 

! 





© © 




© © 




© © 




© © 




*/) 

*4— 




© © 




© © 




© © 




© © 




O 

L_ 

d) 




© © 




© © 


£ 

o 


© © 




© © 




X) 

£ 


v> 


© © 


V> 

a 


© © 


o 

L. 

M- 


© © 


CTl 


© © 




3 

z 


O 

4- 

C 


© © 


+■ 

M— 

*4— 


© © 


O 

4- 

3 


© © 


*4- 

’a 

> 


© © 






o 

V) 

4- 


© © 


O 

V) 

4- 


© © 


O 

V) 

4- 


© © 


V) 

4- 


© © 


.d) 

4^ 




C 

d) 

~o 


© © 


C 

d) 

"O 


© © 


C 

d) 

“O 


© © 


C 

d) 

“O 

- i 


© © 






3 

4- 

V) 


© © 


3 

4- 

V) 


© © 


4- 

V) 

t « 


© © 


m-J 

4- 

V) 

li 


© © 


X 


V) 

0 

1 


M— 

O 

SL 


©© 


*4— 

O 

L_ 


©© 


o 

t_ 


©© 


o 

L_ 


©© 


§ 


1 


d) 

_Q 

£ 


© © 


d) 

-Q 

£ 


© © 


d) 

X 

£ 


© © 


d) 

X 

£ 


© © 


1 




3 

z 




3 

z 




3 

Z 




3 

z 









-a 

Z3 



© © 
© © 
© © 
© © 

© 

© 

0 



£_ 

X 

u 

f— 



© © 
© © 
© © 
© © 
© © 

© 

© 

© 



•O 



155 



Adapted from Nesselrodt & Schaffer 2000. Machine Scannable Version © KDE A AEL 2001. 



Kentucky Extended School Services Program: 
QAIT* Assessment of Classroom 




O 

§ 



.?■ 

§ 

g- 

€ 

.s: 

£ 



'O' 

'o 

I 

~§ 

* 

I 

§ 

5* 

£ 

5 

CO 

€ 

S 
t 
£ 
s~ 

^ ir> 






i 

CO 

.8 

.s: 

S) 

§ 

Jo 



5 

5 



1 


© 


© 


© 


© 


© 


t 

<U 




© 


© 


© 


© 


© 


ZJ 

c 




© 


© 


© 


© 


© 


*4- 

c 




© 


© 


© 


© 


© 


o 

VJ 


ike 

class 


© 


© 


© 


© 


© 





■5-SS 

o c 



“O 

CD 

Z3 

C 

4- 

c 

o 

o 

V / 

c 

.2 

+— 

o 

3 



10 

C 



o 

I* 

o 

a 



§ 

Cj -5 

-c 



SI 

M -S 

•5 53 

O <; 



O X3 

ERLC 



s; >-i 



c 

o 



o 

3 



i/l 

c 



o 

£ 

o 

Of 



£_ 

O 

£ 

3 

_C 



"O 

CD 



U 

CD 

CL 

V) 



O -Z? 



CD 

CO 

c 

<d 

CO 



CO 

2 

o 



CD 

_c 

u 

a 

CD 

4- 

CD 



c_ 

a 

.2 

o 

CD 

£_ 

a 

CO 

CD 

> 

4“ 

u 

CD 

J? 

O 

c 

o 

co 

V) 

CD 



LO 



CD 

-C 

u 

a 

CD 

4- 

CD 



o 

>~ 

“a 

s_ 

o 

CO 
CD 
> 
4— 
U 
CD 
— > 
_Q 
O 
C 

o 

CO 

CO 

.2 

CO 

CD 

4- 

a 

4- 



a 

£ 

S- 

o 

H- 

C 

J_ 

0 
\ 
~D 

§ 

1 



CJl CO 
C 4- 

£ ^ 

I 1 

.£ vj 



c 

CD 

£ 

CO 

CO 

CD 

CO 

CO 

a 



© © © © 
©00© 
© © © © 
© © © © 
© © © © 



CD 

JC 

O 

a 

CD 

4- 

CD 

€ 



c 

CD 

~D 

3 

4— 

CO 

o 

4- 

CD 

CO 

c 

CD 

CO 

CD 

a 

£ 

co 

c 

o 

CO 

CO 

CD 



>. 
a 
2 

>- _2 



CD 

~D 



c 

a 



c 

o 

4— 

O 

£ 

L. 

O 

H- 

C 

CO 

CD 

N 

*C 

a 

cn 



co 

u 

cl 

o 

4- 

2 

CD 

c 



CO 

c 

o 



CO 

c 

a 



CO 

CD 

4- 

O 

Z 



3 



Q_ 

£ 

a 

x 

CD 

"O 

c 

a 

co 

CD 

cn 

o 

£ 

~D 

> 

> 

X 

c 

a 

£ 

co 

CD 

CO 

D 



co 

CD 



o 

c 



C 

CD 

CO 

CO 

CD 

CO 

CD 



CO 

CD 



C 

CD 

3 

cr 

CD 



O "D 



■O 

c 

3 

o 

c_ 

J* 

u 

a 

_Q 

'co 

4- 

c 

CD 

~o 

3 

4— 

CO 

o 

4- 

CD 

4- 

_g 

CD 

£_ 

CO 

c 

o 

CO 

CO 

CD 



CNJ 



CD 

> 



o 

CD 



O 

U 

"O 

c 

o 

CD 

4— 

g 

CD 

£ 

£ 

CO 

CD 

■g 

*> 

o 



u 

a 

-Q 

~D 

<D 

CD 



CD 

O 

a 

CL 

CD 

4- 

g 

‘c_ 

CL 

O 

L. 

CL 

CL 

O 

C 

a 

CD 

co 

3 

co 

L_ 

CD 

_c 

u 

o 



vO 



s_ 

CD 

-C 

u 

a 

CD 

4- 

CD 



© 

© 

© 

© 

0 



CD 

N 

*C 

o 

cn 



"O 

CD 

o 

c 

§ 

T3 

a 

CO 

a 

-C 

u 

3 

CO 

CO 

CD 

U 

’> 

CD 

T3 

CO 

CD 

CO 

D 



© 

© 

© 

© 

0 



"O 

CD 

c 

s_ 

a 

CD 



co 

3 

o 

’> 

CD 



O 

CO 

4- 

c 

CD 

~D 

3 

4— 

CO 

CO 

~o 

c 



JO 

a 



- CD 



O 

£ 



c 

CD 

4- 

c 

o 

u 

c_ 

CD 

> 

o 

o 



© 

© 

© 

© 

0 



£ 

CO 

g 

co 

3 

4- 

c 

CD 

CO 



JT 

X 

CD 

C_ 

CD 

JT 

U 

a 

CD 

4- 

CD 



00 



Cl) 

E 

H 

M- 

O 

(i) 
V > 

D 

"O 

c 

a 

> 

4 — 

C 

fc) 

u 

c 

Ml 

c 

o 



O 

3 



V) 

c 



> 

(X) 

_) 

(X) 

4 - 

a 

's_ 

Q- 

O 

S_ 

CL 

Q- 

<1 

C 

o 

4— 

U 

3 



V) 

c 



o 

>- 

+: 

"a 

a 



V) 

"D 

c 

a 

4 — 

V> 

I- 

M 

< 

a 

•K 



O 

O 

CNJ 

-J 

LU 

< 

LU 

n 

© 

c 

o 

CO 



jQ 

a 



a 

o 

cn 



-C 

u 

a 



O 

O 

O 

CNJ 



o 

cn 



T> 

O 

L 

<D 

CO 

CO 

CD 

z 

£ 

o 



"O 

CD 

4- 

Ql. 

O 

"O 



156 



y> 






$ 0 
■S 

0 ^ 





©©©©©©©© 


© © © 


© 


© 


© © 


.VI 


©©©©©©©© 


© © © 


© 


© 


© © 




©©©©©©©© 


© © © 


© 


© 


© © 




©©©©©©©© 


© © © 


© 


© 


© © 




©©©©©©©© 


© © © 


© 


© 


© © 



■o 

CD 

3 

C 

"■4— 

c 

o 

u 

ft> 

> 



VI 

c 

o 



V) 

ft) 

.> 

'-*— 

c 

ft) 

u 

c 



a — 



V) 

_* 

u 

ft) 

X 

u 

_* 

t_ 

o 

2 

ft) 

£ 

o 

X 



a 

3 

~o 



ft) 

5 

In 

cj 

a 

o 



o 

c 

a 



CO 

ft) 


CO 

c 

Q 




LL 

X 

ft) 


> 

13 


ft) 

c 

0 


VI 

c 

0 


CL 

CO 

ft) 

C_ 

15 


CO 

“O 

C_ 


X 

CD 


C 

X 


ft) 

CD 


CL 

VI 

ft) 

C_ 


a 

2 

ft) 

C_ 


X 

CD 

C 


2 

CO 


8 

3 

O 

CJ 


L_ 

O 

M- 


c_ 

0 

CL 


■O 

c 

a 


a 

CJ 


CL 

3 

O 


§ 

CO 


CD 

C 


CD 

c 


CO 

c 


‘c 

3 

£ 

£ 


£_ 

CD 


■*- 

§ 


+2 

a 


70 


ft) 


*a 

£ 


“O 

3 




‘3 


O 


0 


+- 


§ 




1- 


0 




CO 


"d 


ft) 


H— 


CD 


X 





V) 

ft) 

u 

c 

ft) 

cd 

c 

"-*— 

c 

o 

u 

CL 

3 

o 



a 

L. 

O 

> 

a 

x 

ft) 

x 



CO 

c 



X 

ft) 

V) 

ft) 

V) 

3 

S_ 

ft) 

x 

u 

a 

ft) 

-♦— 

ft) 

x 

H 



V) 

a 

X 
u 

3 

V) 

V) 

ft) 

> 

c 

ft) 
cj 

.£ d -d 



c 

ft) 

£ 

ft) 

> 

o 

s_ 

CL 

£ 



CO 

~D 

s_ 

a 

2 

ft) 

"D 

§ 

2 

ft) 

£ 



V) 

ft) 

cj 

c 

ft) 

cd 

c 

4~ 

c 

o 

u 

CL 

3 

o 



V) 

.1 1 

H ^ 

to w 

e ■£ 

V) _£> 

“O p 



V) 

CO 

ft) 

u 

0 

3 

CO 



~D 

S 



c 

ft) 

"O 

3 

CO 

ft) 



> 

o 



cj 

a 

ft) 

■4— 

ft) 



V) 



0 

M- X) 

d) V) 

1 I 

9 - £ 

£ LLJ 
Q_ 

CL 

a a 



ft) 

§ 

i- 



(D 

£ 



■o 

ft) 

4 - 

a 

u 

o 



~D 

ft) 

a 

u 

_o 

“a 

CO 

ft) 

£ 

v c 
o 

± 4= 

u 

3 



5 ? 

CO 

ft) 

o 

ft) 

z 



CO 

c 



< d 



ft) 

> 



o 

ft) 



CO 
ft) 
V) 
3 

L. 
ft) 
X 
u 

s 

ft) 

CD |E 
O ^ 
CD 

c 

uu a 



v> 

ft) 



~D 

ft) 



V) 

C 

o 

vi 

V) 

ft) 



"O 

c 

ft) 



s 0 

£ £ 

ft) c 

CD ft) 

i - 

E £ 







ft) 

0 

C 






fH 


OJ 


ft) 

00 










M 






▼H 


iH 


— ) -r-l 








v> 

Cj 

.V| 




© 


© 


© © © 


© © 


Q Q 


Q Q 


© © 








© 


© 


© © © 


© © 


Q Q 


© 0 


© © 








© 


© 


© © © 


© © 


Q Q 


Q Q 


© © 








© 


© 


© © © 


© © 


Q Q 


Q Q 


© © 








© 


© 


© © © 


© © 


Q Q 


Q Q 


© © 



•5 S 
0 <: 



c 

o 

4— 

o 

3 

L 

VI 

c 



ft) 

> 

ft) 



ft) 

■H 

’C 

CL 

o 

C_ 

CL 

CL 

•*T 



o 

ERIC 



c 

ft) 

~o 

3 



a 

£ 

V) 

ft) 

CD 

ft) 

a 



a 

c 

o 

CJ 

3 



C 

ft) 



ft) 

ft) M— 

> M- 

-2 -6 



C 

ft) 



c 

ft) 



VI 0) 

c -5 

-O S 

3 ■*- 

V) ft) 

* f 

X) 



CD 

c 

a. 

3 

o 



X 

a 

c 

V) 

3 

O 

ft) 





3 




3 




c 


ft) 


4— 


ft) 

X 


CO 


ft) 


CO 




ft) 

CO 


C 

JD 


X 


u 


CO 


CD 


CO 


CO 

ft) 


ft) 


O 


a 


s 

4— 


ft) 

■*— 

a 


"O 

ft) 


ft) 

a 


CD 

ft) 


C_ 

O 


CO 

CO 


ft) 


"O 


3 


"O 


a 




L. 


0 


0 


0 


0 


c_ 


O 


ft) 


CJ 




£ 

£ 


c 

JC 


£ 

£ 


CD 

C 


L. 

CO 


X 

ft) 


1 

# c 


V) 


0 


L. 


0 


*c 


CD 


CD 


CO 


u 


O 


u 


C_ 


C 


O 


ft) 


.2 

4— 


u 

< 


CL 


CJ 

< 


_8 


CL 

3 


-X 


CO 

Z> 


• — 










O 


C_ 




0 










L. 


0 




a 


d 




X 




O 


2 


d 



V) 

5 b 

1 a 

2 c 

h o 

o C 
CD .2 

c +- 

5 g 

S t; 

14 

9 > 

o "O 



c 

o 



0 

3 



CL 

ft) 

<J 

c 

o 

cj 



O C 



cj 

a 

VI 

a 

X 



o 

cj 

V) 

ft) 

VI 

Z> 



V) 

ft) 

VI 

a 

co 



~D 

C 

a 



VI 



VI 

.c 

~D 

ft) 

N 

“a 

3 

~D 



~D 

C 

VI 

ft) 

VI 

Z) 



X 



vi 

o 



3 

u 



c 

ft) 

■O 

3 



VI 

c 

o 

£ 



ii 

1 ? 



VI 

ft) 

> 



ft) M 

“O -D 



CD 
C 
VI VI 

S Q_ 

1 - 

O VI 

C_ ^ 

a cd 
. c 



ft) 

X 

o 



3 

+- 

VI 

o 

VI 

u 

CL 

o 

CD 

c 



o 

M- 

c 

s_ 

ft) 

> 

o 

u 

VI 

^6 

o 



CD 

c 

t5 

ft) 

L. 

ft) 

■*- 

_>»- 

8 



s ? -g J 



U "O 



00 



ft) 

> 

’•P 

c 

ft) 



ft) 

|S 

CD 



~D 

3 



3 

O 



CD 

C 



VI 

ft) 



< X 



a x o "D 



157 



u 

£ 

ft) 

■D 

O 

u 

O 

u 

CO 

c 



X 

ft) 

VI 

ft) 

VI 

3 



-X 

KJ 

O 

X 

-O 

ft) 

ft) 

M- 

-o 

c 

o 



■*“ c 

a) ft) 

t * 



VI ft) 
ft) VI 

> a 
£_ 
CL 



X 

a 

+- 

c 

3 

o 

u 

u 

< 



a x 






• • ¥ 
£ X 
O 

E -C 

Q_ 

1/1 

8 a 

E o 

&) v> 
— (0 
_ CL 

O T> 



in 



o 

SL 

u 

in 

“O 

<0 

“D 



c 

a 

4 — 

C 

<0 

e 

c 

o 

E 



ft, I 

5 iS 

>- E 
o 

u o 

5 &> 

§ s 

^ u 




<u 



•5 £ 

5 & 

rO -5 

s •§ Vi 
S° 'tj 
^ 5i ^ 
^ § -§ 
$ ^ 

E B 
sfc 

§ ^ 

* S 

.? I 

| ^ 
5 * 



g* 

E 



■L. 

,S: 



<o co 



£ 



V. s sX 
<u *5 v 
~c- x :N 



^ * 

*5 ^ 

*1 § 

^ V) 

« & § 

8 ^ 



S' 

jo 



s: 

.o 

■i^ 

s 

A) 

*o 



S 5 
•S o 

§) § 

*q 5° 

J 5 3 

<X. <0 o 



V) <o 

S 5: 

.0 +- 

■£ . 5 : 

§* J 
§ ^ 

co 0 

« | 

O Cj 



-C 

•2 co 

*1 « 

S r- 'Cl 

« f -8 

8 c ^ 

^ ^ to 

2 ^3 > 

•5 S 

ft) ^ 

I 8 

CO Co 



V> 



+-. 

<0 

CO 

g- 

E 



"O 

CO 

V) 

Z> 



V) 

> 



e 

o 

4 - 

u 

<0 

o* 

e 

CL 

*0 

a 

<0 

-E 

e 

<0 

> 

O 



V) 

O 

o 

-Q 

4— 

X 



e 

o 

\n 

_^j 

£ 



u 

a 

\ 

V) 

-X 

o 

o 

_o 

E 

o 



V) 

a 

e 

CO 

4- 

a 

e 



u 

a 

e 

I 



3 

£ 

H- 

O 

CO 

V) 

D 



e 

CO 

4 - 

3 

CL 

£ 

o 

<0 



OOO 



OOO 



V) 

o 

o 

_Q 



Vi 

4— 

CO 

CO 

-E 

V) 

E 

o 



CO 








v> 

D 


O 


O 


O 


Vis. 


O 


O 


O 



_V) 

a 

e 

CO 

4- 

o 

£ 

4— 

V) 

X 

CO 

V) 

t 

e 

o 

e 

H- 

o 

CO 

V) 

D 



V) 
CO 
> 

a 
3 

Q- 
*C 

a 
£ 



co 

*0 

3 

4 — 

cn 



O 

O 



cn 

cn 

_o 

on 

c 

‘c 

L_ 

a 

\ 

jo 

a 

c 

s_ 

3 

O 



V) 

a 

s_ 

CO 

4 - 

a 

£ 

c 

0 

1 

V) 

*0 

c 

a 



>- 

s_ 

a 

s_ 

£ 

o 

o 

1_ 

V) 

V) 

_a 

U 



£ 

o 

o 

s_ 

V) 



V) 

CO 

o 

c_ 

3 

o 

V) 

CO 

S- 

o 

3 

< 



o 

o 



V) 

a 

*s_ 

CO 

4- 

a 

£ 

CO 

o 

E 

CO 

£_ 

CO 

H- 

CO 



V) 

s_ 

(0 

>* 

a 



V) 

£> 

u 

vT 

co 

a_ 

o 



OOO 

OOO 



V) 

co 

o 

E 

3 

O 

V) 

CO 

£- 

O 

CO 

*0 



“O 

c 

a 

CL 

a 



~o 

c 

a 

V) 

CO 

£ 

a 



O O 
O O 



V) 

I_ 

CO 

s 

o 



V) 

o 

V) 



V) 

CO 

CL 

o 



co .r 

o 

o 



V) 

N 

N 

3 

CL 

S_ 

o 



V) 

CL 

O 

s_ 

CL 

\ 

V) 

“O 



co 

-O 
on ^ 



a 

E 

O 

4— 

O 

3 

£_ 

4— 

V) 

E 



o 

o 



co 

E 

3 

4- 

a 

E 

CO 

a. 

£ 

CO 



O O 







V) 

ri 


V) 




-y— 




v> 

JO 

3 

E 

£ 






E 




E 


•O 




u 


CO 




O 


CO 


V) 

4- 

E 

CO 


an 

E 

4- 

*> 


4 - 

E 

CO 

0 


on 

E 


4- 

_a 

■P 

E 


>- 

_a 

CL 

V) 


0 

0 

E 


£ 

E 


E 


4 - 

*> 


*4- 

-E 


CO 

> 


~o 


V) 

V) 

_a 


an 

V) 

Vi 


"D 

E 

a 


+“ 

0 

a 


an 

co 


CO 

-Q 

a 


E 

O 

3 


CJ 


a 


"3 


4— 


4 - 


4- 


4 - 


*0 

CO 

4— 

V) 

0 


*0 

co 

4 — 

V) 

0 


H- 

E 

CO 

CO 

.E 


U 

E 

+“ 

V) 


a 

3 

a- 

co 

"O 


E 

O 

H- 

£ 

0 


E 

CO 

*0 

3 

4- 


CL 


CL 




b 


< 


VJ 


E0 


O 


O 


O 


O 


O 


O 


O 



o 

o 

00 

_) 

LU 

< 

rvX 



o 

ERIC 



158 



No distracting internal noises/interruptions O O Science/lab table(s) 

No distracting external noises/interruptions O O Classroom chalkboard 

Open, risk-free environment O O Student-used equipment 



Appendix H: 



School and Program Description Form 



Exf e , 



Kentucky Extended School Services Program: 
School and Program Description Form 




Please provide a description of your school and your ESS program by filling in your responses 
to the items below. Fill in response circles completely, like this: • 



1. I am the: 

O ESS coordinator 
O ESS coordinator/principal 
O ESS coordinator/teacher 
O Principal/vice principal 
O Classroom teacher 
O Other position 

2. The school is a: 

O Elementary school 
O Middle school 
O High school 
O Other configuration 

3. The ESS site is: 

O School-based 
O Community-based 

4. The community is: 

O Rural 
O Suburban 
O Urban 



5. Total student enrollment in the school: 

(ex.: 79 students would be coded as 0079 
by coding one digit per row of bubbles) 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

6. Total student enrollment in ESS program: 

©o©®©©©©©® 

©o©®©©©©©® 

©o©®©©©©©® 

7. Total number of ESS teachers: 

©o©®©©©©©® 

©o©®©©©©©® 

8. Average size of ESS class: 

©o©®©©©©©® 

©o©®©©©©©® 

9. Total number of ESS teachers who are also 
regular classroom teachers: 

©o©®©©©©©® 

©o©®©©©©©® 

Continue 



Adapted from Nesselrodt A Schaffer 2000. Machine Scannable Version © KDE A AEL 2001. 
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10. Number of days per week program operates: ©o©®©©®© 

11. Number of hours per day program operates: ©©©©©©©©© 

12. The ESS program operates: (select all that apply) 

O Before school O Evenings O Summer 

O After school O Weekends O Intersession(s) 

13. When did your ESS program originally start? 



o 


1990 


O 


1993 


0 


1996 


0 


1999 


o 


1991 


O 


1994 


0 


1997 


0 


2000 


o 


1992 


O 


1995 


0 


1998 


0 


2001 



(For scanning purposes, please keep your responses to the following items inside each box.) 

14. Describe the major components of your ESS program and the current level of 
implementation for each component. 




15. Describe any unique characteristics of your community, school, or student population. 



Appendix I: 

Innovation Component Configuration Map 
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KENTUCKY EXTENDED SCHOOL SERVICES PROGRAM: 

Innovation Component Conf iguration Map 
Demographic Scoring Page 
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Adapted from Kentucky Institute for Education Research 1996. Machine Scannabie Version ® KDE d AEL 2001. 
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District-wide ESS Program Planning 

1. Collaborative Planning Processes 

2. Program Evaluation 

3. Fiscal Management 

4. Linkages With Other KERA Strands and Other Supporting Programs 



Kentucky Extended School Services Program: 
Innovation Component Conf iguration Map 
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. Fiscal Management 

. Linkages With Other KERA Strands and Other Supporting Programs 
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BEST COPY AVAILABLE 



Students are referred by a teacher, Students are referred by a teacher, Students are referred and placed in 

parent, or student. Students enter parent, or student. Students remain the program without clearly defined 

and exit a program as need becomes in the program longer than needed. goals and remain in the program the 

evident and clearly identified goals entire school year, regardless of need, 

are met. 
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content, morning, afternoon, night, conflicts with extracurricular 

Saturday, summer, activities and transportation con- 

straints may influence ESS schedule. 
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SBDM councils) collaborate to deter- dinators and principals) determine programs will start and end, the number 

mine which schools will offer services, which schools will offer services, of teachers, and whether or not trans- 

when programs will start and end, the when programs will start and end, the portation will be provided, 

number of teachers, and whether or number of teachers, and whether or 
not transportation will be provided. not transportation will be provided. 
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